This is a trivial addition to the SG API that can receive kernel
pointers. It is only used by the out-of-tree test module. So
it's immediate need is questionable. For maintenance ease it might
just get in, as it's very small.

John.
do you need this in the Kernel, or is it only for osd_ktest.ko?
Signed-off-by: NJohn A. Chandy <john.chandy@uconn.edu>
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

This patch adds the Scatter-Gather (sg) API to libosd.
Scatter-gather enables a write/read of multiple none-contiguous
areas of an object, in a single call. The extents may overlap
and/or be in any order.

The Scatter-Gather list is sent to the target in what is called
a "cdb continuation segment". This is yet another possible segment
in the osd-out-buffer. It is unlike all other segments in that it
sits before the actual "data" segment (which until now was always
first), and that it is signed by itself and not part of the data
buffer. This is because the cdb-continuation-segment is considered
a spill-over of the CDB data, and is therefor signed under
OSD_SEC_CAPKEY and higher.

TODO: A new osd_finalize_request_ex version should be supplied so
the @caps received on the network also contains a size parameter
and can be spilled over into the "cdb continuation segment".

Thanks to John Chandy <john.chandy@uconn.edu> for the original
code, and investigations. And the implementation of SG support
in the osd-target.
Original-coded-by: NJohn Chandy <john.chandy@uconn.edu>
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

At osd_end_request first free the request that might
point to pages, then free these pages. In reverse order
of allocation. For now it's just anal neatness. When we'll
use mempools It'll also pay in performance.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

The _osd_req_finalize_attr_page was off by a mile, when trying to
append the enc_get_attr segment instead of the proper set_attr segment.

Also properly support when we don't have any attribute to set while
getting a full page. And when clearing an attribute by setting it's
size to zero.
Reported-by: NJohn Chandy <john.chandy@uconn.edu>
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

Return of the bi_rw tests is no longer bool after commit 74450be1. So
testing against constants doesn't make sense anymore. Fix this bug in
osd_req_read by removing "== 1" in test.

This is not a problem now, where REQ_WRITE is 1, but this can change
in the future and we don't want to rely on that.
Signed-off-by: NJiri Slaby <jslaby@suse.cz>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

Remove the current bio flags and reuse the request flags for the bio, too.
This allows to more easily trace the type of I/O from the filesystem
down to the block driver.  There were two flags in the bio that were
missing in the requests:  BIO_RW_UNPLUG and BIO_RW_AHEAD.  Also I've
renamed two request flags that had a superflous RW in them.

Note that the flags are in bio.h despite having the REQ_ name - as
blkdev.h includes bio.h that is the only way to go for now.
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NJens Axboe <jaxboe@fusionio.com>

include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h

percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files.  percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.

percpu.h -> slab.h dependency is about to be removed.  Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability.  As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.

  http://userweb.kernel.org/~tj/misc/slabh-sweep.py

The script does the followings.

* Scan files for gfp and slab usages and update includes such that
  only the necessary includes are there.  ie. if only gfp is used,
  gfp.h, if slab is used, slab.h.

* When the script inserts a new include, it looks at the include
  blocks and try to put the new include such that its order conforms
  to its surrounding.  It's put in the include block which contains
  core kernel includes, in the same order that the rest are ordered -
  alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
  doesn't seem to be any matching order.

* If the script can't find a place to put a new include (mostly
  because the file doesn't have fitting include block), it prints out
  an error message indicating which .h file needs to be added to the
  file.

The conversion was done in the following steps.

1. The initial automatic conversion of all .c files updated slightly
   over 4000 files, deleting around 700 includes and adding ~480 gfp.h
   and ~3000 slab.h inclusions.  The script emitted errors for ~400
   files.

2. Each error was manually checked.  Some didn't need the inclusion,
   some needed manual addition while adding it to implementation .h or
   embedding .c file was more appropriate for others.  This step added
   inclusions to around 150 files.

3. The script was run again and the output was compared to the edits
   from #2 to make sure no file was left behind.

4. Several build tests were done and a couple of problems were fixed.
   e.g. lib/decompress_*.c used malloc/free() wrappers around slab
   APIs requiring slab.h to be added manually.

5. The script was run on all .h files but without automatically
   editing them as sprinkling gfp.h and slab.h inclusions around .h
   files could easily lead to inclusion dependency hell.  Most gfp.h
   inclusion directives were ignored as stuff from gfp.h was usually
   wildly available and often used in preprocessor macros.  Each
   slab.h inclusion directive was examined and added manually as
   necessary.

6. percpu.h was updated not to include slab.h.

7. Build test were done on the following configurations and failures
   were fixed.  CONFIG_GCOV_KERNEL was turned off for all tests (as my
   distributed build env didn't work with gcov compiles) and a few
   more options had to be turned off depending on archs to make things
   build (like ipr on powerpc/64 which failed due to missing writeq).

   * x86 and x86_64 UP and SMP allmodconfig and a custom test config.
   * powerpc and powerpc64 SMP allmodconfig
   * sparc and sparc64 SMP allmodconfig
   * ia64 SMP allmodconfig
   * s390 SMP allmodconfig
   * alpha SMP allmodconfig
   * um on x86_64 SMP allmodconfig

8. percpu.h modifications were reverted so that it could be applied as
   a separate patch and serve as bisection point.

Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: NTejun Heo <tj@kernel.org>
Guess-its-ok-by: NChristoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>

Doing CHECK="smatch --two-passes gives:
drivers/scsi/osd/osd_initiator.c +1435 osd_finalize_request warning: assignment to 'ret' was never used

Which is an unchecked possible allocation failure, Fixed.
Reported-by: NDan Carpenter <error27@gmail.com>
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

So libosd has decided to sacrifice some code simplicity for the sake of
a clean API. One of these things is the possibility for users to call
osd_end_request, in any condition at any state. This opens up some
problems with calling blk_put_request when out-side of the completion
callback but calling __blk_put_request when detecting a from-completion
state.

The current hack was working just fine until exofs decided to operate on
all devices in parallel and wait for the sum of the requests, before
deallocating all osd-requests at once. There are two new possible cases
1. All request in a group are deallocated as part of the last request's
   async-done, request_queue is locked.
2. All request in a group where executed asynchronously, but
   de-allocation was delayed to after the async-done, in the context of
   another thread. Async execution but request_queue is not locked.

The solution I chose was to separate the deallocation of the osd_request
which has the information users need, from the deallocation of the
internal(2) requests which impose the locking problem. The internal
block-requests are freed unconditionally inside the async-done-callback,
when we know the queue is always locked. If at osd_end_request time we
still have a bock-request, then we know it did not come from within an
async-done-callback and we can call the regular blk_put_request.

The internal requests were used for carrying error information after
execution. This information is now copied to osd_request members for
later analysis by user code.

The external API and behaviour was unchanged, except now it really
supports what was previously advertised.
Reported-by: NVineet Agarwal <checkout.vineet@gmail.com>
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Cc: Stable Tree <stable@kernel.org>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

Administer some love to the osd_req_decode_sense function

* Fix a bad bug with osd_req_decode_sense(). If there was no scsi
  residual, .i.e the request never reached the target, then all the
  osd_sense_info members where garbage.

* Add grossly missing in/out_resid to osd_sense_info and fill them in
  properly.

* Define an osd_err_priority enum which divides the possible errors into
  7 categories in ascending severity. Each category is also assigned a
  Linux return code translation.

  Analyze the different osd/scsi/block returned errors and set the
  proper osd_err_priority and Linux return code accordingly.

* extra check a few situations so not to get stuck with inconsistent
  error view. Example an empty residual with an error code, and other
  places ...

Lots of libosd's osd_req_decode_sense clients had this logic in some
form or another. Consolidate all these into one place that should
actually know about osd returns. Thous translating it to a more
abstract error.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

When an error was detected in an attribute list do to
a target bug. We would print an error but spin endlessly
regardless. Fix it.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

The (never tested) osd_sense_attribute_identification case
has never worked. The loop was never advanced on.
Fix it to work as intended.

On 10/30/2009 04:39 PM, Roel Kluin wrote:
  I found this by code analysis, searching for while
  loops that test a local variable, but do not modify
  the variable.
Reported-by: NRoel Kluin <roel.kluin@gmail.com>
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

Define an osd_dev_info structure that Uniquely identifies an OSD
device lun on the network. The identification is built from unique
target attributes and is the same for all network/SAN machines.

osduld_info_lookup() - NEW
    New API that will lookup an osd_dev by its osd_dev_info.
    This is used by pNFS-objects for cross network global device
    identification. And by exofs multy-device support, the device
    info is specified in the on-disk exofs device table.

osduld_device_info() - NEW
    Given an osd_dev handle returns its associated osd_dev_info.
    The ULD fetches this information at startup and hangs it on
    each OSD device. (This is a fast operation that can be called
    at any condition)

osduld_device_same() - NEW
    With a given osd_dev at one hand and an osd_dev_info
    at another, we would like to know if they are the same
    device.
    Two osd_dev handles can be checked by:
        osduld_device_same(od1, osduld_device_info(od2));

osd_auto_detect_ver() - REVISED
    Now returns an osd_dev_info structure. Is only called once
    by ULD as before. See added comments for how to use.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@suse.de>

libosd has it's own sense decoding and printout. Don't
let scsi_lib duplicate that printout. (Which is done wrong
in regard to osd commands)
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

libosd users that need to work with bios, must sometime use
the request_queue associated with the osd_dev. Make a wrapper for
that, and convert all in-tree users.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

For supporting of chained-bios we can not inspect the first
bio only, as before. Caller shall pass the total length of the
request, ie. sum_bytes(bio-chain).

Also since the bio might be a chain we don't set it's direction
on behalf of it's callers. The bio direction should be properly
set prior to this call. So fix a couple of write users that now
need to set the bio direction properly

[In this patch I change both library code and user sites at
 exofs, to make it easy on integration. It should be submitted
 via James's scsi-misc tree.]
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
CC: Jeff Garzik <jeff@garzik.org>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

_osd_req_finalize_data_integrity was trying to deduce the number of
out_bytes from passed osd_request->out.bio. This is wrong when
the bio is chained. The caller of _osd_req_finalize_data_integrity
has more ready available information and should just pass it.

Also in the light of future support for CDB-continuation segment this is
a better solution.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

By popular demand, define usefull wrappers for osd_req_read/write
that recieve kernel pointers. All users had their own.

Also remove these from exofs
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Shorten out the Attributes names.
Align all results on column 24.
Print system ID in a new line.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Use new blk_make_request() to allocate a request from bio
and avoid using deprecated blk_rq_append_bio().

This patch is dependent on a block layer patch titled:
    [BLOCK] New blk_make_request() takes bio returns request

This is the last usage of blk_rq_append_bio in osd, it can now
be un-exported.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
CC: Jeff Garzik <jeff@garzik.org>
CC: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Signed-off-by: NJens Axboe <jens.axboe@oracle.com>

Now that blk_rq_map_kern will append the buffer onto the
request we can use it easily for adding extra segments
(eg. attributes)

This patch is dependent on a block layer patch titled:
   [BLOCK] allow blk_rq_map_kern to append to requests
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJens Axboe <jens.axboe@oracle.com>

With recent unification of fields, it's now guaranteed that
rq->data_len always equals blk_rq_bytes().  Convert all non-IDE direct
users to accessors.  IDE will be converted in a separate patch.

Boaz: spotted incorrect data_len/resid_len conversion in osd.

[ Impact: convert direct rq->data_len usages to blk_rq_bytes() ]
Signed-off-by: NTejun Heo <tj@kernel.org>
Acked-by: NSergei Shtylyov <sshtylyov@ru.mvista.com>
Cc: Pete Zaitcev <zaitcev@redhat.com>
Cc: Eric Moore <Eric.Moore@lsi.com>
Cc: Markus Lidel <Markus.Lidel@shadowconnect.com>
Cc: Darrick J. Wong <djwong@us.ibm.com>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>
Cc: Eric Moore <Eric.Moore@lsi.com>
Cc: Boaz Harrosh <bharrosh@panasas.com>
Cc: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Signed-off-by: NJens Axboe <jens.axboe@oracle.com>

OSC's OSD2 target: [git clone git://git.open-osd.org/osc-osd/ master]
(Initiator code prior to this patch must use: "git checkout CDB_VER_OSD2r01"
 in the target tree above)

This is a summery of the wire changes:

 * OSDv2_ADDITIONAL_CDB_LENGTH == 192 => 228 (Total CDB is now 236 bytes)
 * Attributes List Element Header grew, so attribute values are 8 bytes
   aligned.
 * Cryptographic keys and signatures are 20 => 32
 * Few new definitions.

(Still missing new standard definitions attribute values, these do not change
 wire format and will be added later when needed)
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

In OSD2r04 draft, cryptographic key size changed to 32 bytes from
OSD1's 20 bytes. This causes a couple of on-the-wire structures
to change, including the CDB.

In this patch the OSD1/OSD2 handling is separated out in regard
to affected structures, but on-the-wire is still the same. All
on the wire changes will be submitted in one patch for bisect-ability.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

In OSD2r05 draft each attribute list element header was changed
so attribute-value would be 8 bytes aligned. In OSD2r01-r04
it was aligned on 2 bytes. (This is because in OSD2r01 the complete
element was 8 bytes padded at end but the header was not adjusted
and caused permanent miss-alignment.)

OSD1 elements are not padded and might be or might not be aligned.
OSD1 is still supported.

In this code we do all the code re-factoring to separate OSD1/OSD2
differences but do not change actual wire format. All wire format
changes will happen in one patch later, for bisect-ability.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

bio_map_kern() returns an ERR_PTR() not NULL.

Found by smatch (http://repo.or.cz/w/smatch.git).  Compile tested.
Signed-off-by: NDan Carpenter <error27@gmail.com>
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

A fix for a very serious and stupid bug in osd_initiator. It
used to call blk_put_request() regardless of if it was from
the end_io callback or if called after a sync execution.
It should call the unlocked version __blk_put_request() instead.

Also fixed is the remove of _abort_unexecuted_bios hack, and use of
blk_end_request(,-ERROR,) to deallocate half baked requests. I've
audited the code and it should be safe.

Reported and
Tested-by: NXu Yang <onlyxuyang@qq.com>
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Very old OSC's Target had a BUG in the Get/Set attributes where
it was looking in the wrong places for attribute lists length.
If used with the open-osd initiator, the initiator would dereference
a NULL pointer when retrieving system_information attributes.

Checks are added that retrieval of each attribute is successful
before accessing its value.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Implementation of the osd_req_decode_sense() API. Can be called by
library users to decode what failed in command executions.

Add SCSI_OSD_DPRINT_SENSE Kconfig variable. Possible values are:
0 - Do not print any errors to messages file <KERN_ERR>
1 - (Default) Print only decoded errors that are not recoverable.
    Recoverable errors are those that the target has complied with
    the request but with a warning. For example read passed end of
    object will return zeros after the last valid byte.
2- Print all errors.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Auto detect an OSDv2 or OSDv1 target at run time. Note how none
of the OSD API calls change. The tests do not know what device
version it is.

This test now passes against both the IBM-OSD-SIM OSD1 target
as well as OSC's OSD2 target.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Reviewed-by: NBenny Halevy <bhalevy@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Add support for OSD2 at run time. It is now possible to run with
both OSDv1 and OSDv2 targets at the same time. The actual detection
should be preformed by the security manager, as the version is encoded
in the capability structure.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Reviewed-by: NBenny Halevy <bhalevy@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Some commands declared in header are not yet implemented. Put them
as stubs in .c file, just so they take their place in the file
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Reviewed-by: NBenny Halevy <bhalevy@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Add support for the various List-objects commands. List-partitions-in-device,
List-collections-in-partition, List-objects-in-partition,
List-objects-in-collection. All these support partial listing and continuation.

Add support for the different Flush commands and options.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Reviewed-by: NBenny Halevy <bhalevy@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Layout the signing of OSD's CDB and all-data security modes. The actual
code for signing the data and CDB is missing, but the code flow and the extra
buffer segments are all in place.
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Reviewed-by: NBenny Halevy <bhalevy@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Support for both List-Mode and Page-Mode osd attributes. One of
these operations may be added to most other operations.

Define the OSD standard's attribute pages constants and structures
(osd_attributes.h)
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Reviewed-by: NBenny Halevy <bhalevy@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>

Implementation of the most basic OSD functionality and
infrastructure. Mainly Format, Create/Remove Partition,
Create/Remove Object, and read/write.

- Add Makefile and Kbuild to compile libosd.ko
- osd_initiator.c Implementation file for osd_initiator.h
  and osd_sec.h APIs
- osd_debug.h - Some kprintf macro definitions
Signed-off-by: NBoaz Harrosh <bharrosh@panasas.com>
Reviewed-by: NBenny Halevy <bhalevy@panasas.com>
Signed-off-by: NJames Bottomley <James.Bottomley@HansenPartnership.com>