when starting lock mastery (excepting the recovery lock) wait on any nodes
needing recovery. fix one instance where lock resources were left attached
to the recovery list after recovery completed.  ensure that the node_down
code is run uniformly regardless of which node found the dead node first.
Signed-off-by: NKurt Hackel <kurt.hackel@oracle.com>
Signed-off-by: NMark Fasheh <mark.fasheh@oracle.com>

Switch from list_head to hlist_head. Make the size of the hash dependent
upon the allocated area, rather than a constant.
Signed-off-by: NMark Fasheh <mark.fasheh@oracle.com>

* add dlm_wait_for_node_death function to be used after receiving a network
  error.  this will wait for the given timeout to allow the heartbeat
  callbacks to update the domain map.  without this, some paths may spin
  and consume enough cpu that the heartbeat gets starved and never updates.
Signed-off-by: NKurt Hackel <kurt.hackel@oracle.com>
Signed-off-by: NMark Fasheh <mark.fasheh@oracle.com>

* after successfully taking the $RECOVERY lock in EX mode, recheck to make
  sure that recovery has not already begun or completed on another node
Signed-off-by: NKurt Hackel <kurt.hackel@oracle.com>
Signed-off-by: NMark Fasheh <mark.fasheh@oracle.com>

fs/ocfs2/dlm/dlmrecovery.c does now use msleep(), and does therefore
need to #include <linux/delay.h> for getting the prototype of this
function.
Signed-off-by: NAdrian Bunk <bunk@stusta.de>
Signed-off-by: NMark Fasheh <mark.fasheh@oracle.com>

* fix a hang which can occur during shutdown migration
* do not allow nodes to join during recovery
* when restarting lock mastery, do not ignore nodes which come up
* more than one node could become recovery master, fix this
* sleep to allow some time for heartbeat state to catch up to network
* extra debug info for bad recovery state problems
* make DLM_RECO_NODE_DATA_DONE a valid state for non-master recovery nodes
* prune all locks from dead nodes on $RECOVERY lock resources
* do NOT automatically add new nodes to mle nodemaps until they have properly
  joined the domain
* make sure dlm_pick_recovery_master only exits when all nodes have synced
* properly handle dlmunlock errors in dlm_pick_recovery_master
* do not propagate network errors in dlm_send_begin_reco_message
* dead nodes were not being put in the recovery map sometimes, fix this
* dlmunlock was failing to clear the unlock actions on DLM_DENIED
Signed-off-by: NKurt Hackel <kurt.hackel@oracle.com>
Signed-off-by: NMark Fasheh <mark.fasheh@oracle.com>

A distributed lock manager built with the cluster file system use case
in mind. The OCFS2 dlm exposes a VMS style API, though things have
been simplified internally. The only lock levels implemented currently
are NLMODE, PRMODE and EXMODE.
Signed-off-by: NMark Fasheh <mark.fasheh@oracle.com>
Signed-off-by: NKurt Hackel <kurt.hackel@oracle.com>