1. 21 5月, 2014 1 次提交
    • J
      blk-mq: allow changing of queue depth through sysfs · e3a2b3f9
      Jens Axboe 提交于
      For request_fn based devices, the block layer exports a 'nr_requests'
      file through sysfs to allow adjusting of queue depth on the fly.
      Currently this returns -EINVAL for blk-mq, since it's not wired up.
      Wire this up for blk-mq, so that it now also always dynamic
      adjustments of the allowed queue depth for any given block device
      managed by blk-mq.
      Signed-off-by: NJens Axboe <axboe@fb.com>
      e3a2b3f9
  2. 20 5月, 2014 6 次提交
  3. 19 5月, 2014 1 次提交
  4. 14 5月, 2014 1 次提交
    • J
      blk-mq: improve support for shared tags maps · 0d2602ca
      Jens Axboe 提交于
      This adds support for active queue tracking, meaning that the
      blk-mq tagging maintains a count of active users of a tag set.
      This allows us to maintain a notion of fairness between users,
      so that we can distribute the tag depth evenly without starving
      some users while allowing others to try unfair deep queues.
      
      If sharing of a tag set is detected, each hardware queue will
      track the depth of its own queue. And if this exceeds the total
      depth divided by the number of active queues, the user is actively
      throttled down.
      
      The active queue count is done lazily to avoid bouncing that data
      between submitter and completer. Each hardware queue gets marked
      active when it allocates its first tag, and gets marked inactive
      when 1) the last tag is cleared, and 2) the queue timeout grace
      period has passed.
      Signed-off-by: NJens Axboe <axboe@fb.com>
      0d2602ca
  5. 11 5月, 2014 5 次提交
  6. 10 5月, 2014 3 次提交
    • J
      block: only calculate part_in_flight() once · 7276d02e
      Jens Axboe 提交于
      We first check if we have inflight IO, then retrieve that
      same number again. Usually this isn't that costly since the
      chance of having the data dirtied in between is small, but
      there's no reason for calling part_in_flight() twice.
      Signed-off-by: NJens Axboe <axboe@fb.com>
      7276d02e
    • J
      blk-mq: fix race in IO start accounting · cf4b50af
      Jens Axboe 提交于
      Commit c6d600c6 opened up a small race where we could attempt to
      account IO completion on a request, racing with IO start accounting.
      Fix this up by ensuring that we've accounted for IO start before
      inserting the request.
      Signed-off-by: NJens Axboe <axboe@fb.com>
      cf4b50af
    • J
      blk-mq: use sparser tag layout for lower queue depth · 59d13bf5
      Jens Axboe 提交于
      For best performance, spreading tags over multiple cachelines
      makes the tagging more efficient on multicore systems. But since
      we have 8 * sizeof(unsigned long) tags per cacheline, we don't
      always get a nice spread.
      
      Attempt to spread the tags over at least 4 cachelines, using fewer
      number of bits per unsigned long if we have to. This improves
      tagging performance in setups with 32-128 tags. For higher depths,
      the spread is the same as before (BITS_PER_LONG tags per cacheline).
      Signed-off-by: NJens Axboe <axboe@fb.com>
      59d13bf5
  7. 09 5月, 2014 3 次提交
    • J
      blk-mq: implement new and more efficient tagging scheme · 4bb659b1
      Jens Axboe 提交于
      blk-mq currently uses percpu_ida for tag allocation. But that only
      works well if the ratio between tag space and number of CPUs is
      sufficiently high. For most devices and systems, that is not the
      case. The end result if that we either only utilize the tag space
      partially, or we end up attempting to fully exhaust it and run
      into lots of lock contention with stealing between CPUs. This is
      not optimal.
      
      This new tagging scheme is a hybrid bitmap allocator. It uses
      two tricks to both be SMP friendly and allow full exhaustion
      of the space:
      
      1) We cache the last allocated (or freed) tag on a per blk-mq
         software context basis. This allows us to limit the space
         we have to search. The key element here is not caching it
         in the shared tag structure, otherwise we end up dirtying
         more shared cache lines on each allocate/free operation.
      
      2) The tag space is split into cache line sized groups, and
         each context will start off randomly in that space. Even up
         to full utilization of the space, this divides the tag users
         efficiently into cache line groups, avoiding dirtying the same
         one both between allocators and between allocator and freeer.
      
      This scheme shows drastically better behaviour, both on small
      tag spaces but on large ones as well. It has been tested extensively
      to show better performance for all the cases blk-mq cares about.
      Signed-off-by: NJens Axboe <axboe@fb.com>
      4bb659b1
    • C
      blk-mq: initialize struct request fields individually · af76e555
      Christoph Hellwig 提交于
      This allows us to avoid a non-atomic memset over ->atomic_flags as well
      as killing lots of duplicate initializations.
      Signed-off-by: NChristoph Hellwig <hch@lst.de>
      Signed-off-by: NJens Axboe <axboe@fb.com>
      af76e555
    • J
      blk-mq: update a hotplug comment for grammar · 9fccfed8
      Jens Axboe 提交于
      Signed-off-by: NJens Axboe <axboe@fb.com>
      9fccfed8
  8. 08 5月, 2014 1 次提交
  9. 03 5月, 2014 3 次提交
  10. 01 5月, 2014 3 次提交
  11. 30 4月, 2014 1 次提交
    • J
      blk-mq: fix waiting for reserved tags · 5810d903
      Jens Axboe 提交于
      blk_mq_wait_for_tags() is only able to wait for "normal" tags,
      not reserved tags. Pass in which one we should attempt to get
      a tag for, so that waiting for reserved tags will work.
      
      Reserved tags are used for internal commands, which are usually
      serialized. Hence no waiting generally takes place, but we should
      ensure that it actually works if users need that functionality.
      Signed-off-by: NJens Axboe <axboe@fb.com>
      5810d903
  12. 28 4月, 2014 1 次提交
  13. 25 4月, 2014 2 次提交
    • C
      block: fold __blk_add_timer into blk_add_timer · c4a634f4
      Christoph Hellwig 提交于
      Signed-off-by: NChristoph Hellwig <hch@lst.de>
      Signed-off-by: NJens Axboe <axboe@fb.com>
      c4a634f4
    • C
      blk-mq: respect rq_affinity · 38535201
      Christoph Hellwig 提交于
      The blk-mq code is using it's own version of the I/O completion affinity
      tunables, which causes a few issues:
      
       - the rq_affinity sysfs file doesn't work for blk-mq devices, even if it
         still is present, thus breaking existing tuning setups.
       - the rq_affinity = 1 mode, which is the defauly for legacy request based
         drivers isn't implemented at all.
       - blk-mq drivers don't implement any completion affinity with the default
         flag settings.
      
      This patches removes the blk-mq ipi_redirect flag and sysfs file, as well
      as the internal BLK_MQ_F_SHOULD_IPI flag and replaces it with code that
      respects the queue-wide rq_affinity flags and also implements the
      rq_affinity = 1 mode.
      
      This means I/O completion affinity can now only be tuned block-queue wide
      instead of per context, which seems more sensible to me anyway.
      Signed-off-by: NChristoph Hellwig <hch@lst.de>
      Signed-off-by: NJens Axboe <axboe@fb.com>
      38535201
  14. 24 4月, 2014 3 次提交
  15. 23 4月, 2014 2 次提交
  16. 22 4月, 2014 4 次提交