1. 13 11月, 2013 1 次提交
    • P
      block: Use u64_stats_init() to initialize seqcounts · 90d3839b
      Peter Zijlstra 提交于
      Now that seqcounts are lockdep enabled objects, we need to explicitly
      initialize runtime allocated seqcounts so that lockdep can track them.
      
      Without this patch, Fengguang was seeing:
      
        [    4.127282] INFO: trying to register non-static key.
        [    4.128027] the code is fine but needs lockdep annotation.
        [    4.128027] turning off the locking correctness validator.
        [    4.128027] CPU: 0 PID: 96 Comm: kworker/u4:1 Not tainted 3.12.0-next-20131108-10601-gbad570d #2
        [    4.128027] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
        [    ...     ]
        [    4.128027] Call Trace:
        [    4.128027]  [<7908e744>] ? console_unlock+0x353/0x380
        [    4.128027]  [<79dc7cf2>] dump_stack+0x48/0x60
        [    4.128027]  [<7908953e>] __lock_acquire.isra.26+0x7e3/0xceb
        [    4.128027]  [<7908a1c5>] lock_acquire+0x71/0x9a
        [    4.128027]  [<794079aa>] ? blk_throtl_bio+0x1c3/0x485
        [    4.128027]  [<7940658b>] throtl_update_dispatch_stats+0x7c/0x153
        [    4.128027]  [<794079aa>] ? blk_throtl_bio+0x1c3/0x485
        [    4.128027]  [<794079aa>] blk_throtl_bio+0x1c3/0x485
        ...
      
      Use u64_stats_init() for all affected data structures, which initializes
      the seqcount.
      Reported-and-Tested-by: NFengguang Wu <fengguang.wu@intel.com>
      Cc: Vivek Goyal <vgoyal@redhat.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      Signed-off-by: NPeter Zijlstra <peterz@infradead.org>
      [ Folded in another fix from the mailing list as well as a fix to that fix. Tweaked commit message. ]
      Signed-off-by: NJohn Stultz <john.stultz@linaro.org>
      Signed-off-by: NPeter Zijlstra <peterz@infradead.org>
      Link: http://lkml.kernel.org/r/1384314134-6895-1-git-send-email-john.stultz@linaro.org
      [ So I actually think that the two SOBs from PeterZ are the right depiction of the patch route. ]
      Signed-off-by: NIngo Molnar <mingo@kernel.org>
      90d3839b
  2. 09 8月, 2013 3 次提交
    • T
      cgroup: make css_for_each_descendant() and friends include the origin css in the iteration · bd8815a6
      Tejun Heo 提交于
      Previously, all css descendant iterators didn't include the origin
      (root of subtree) css in the iteration.  The reasons were maintaining
      consistency with css_for_each_child() and that at the time of
      introduction more use cases needed skipping the origin anyway;
      however, given that css_is_descendant() considers self to be a
      descendant, omitting the origin css has become more confusing and
      looking at the accumulated use cases rather clearly indicates that
      including origin would result in simpler code overall.
      
      While this is a change which can easily lead to subtle bugs, cgroup
      API including the iterators has recently gone through major
      restructuring and no out-of-tree changes will be applicable without
      adjustments making this a relatively acceptable opportunity for this
      type of change.
      
      The conversions are mostly straight-forward.  If the iteration block
      had explicit origin handling before or after, it's moved inside the
      iteration.  If not, if (pos == origin) continue; is added.  Some
      conversions add extra reference get/put around origin handling by
      consolidating origin handling and the rest.  While the extra ref
      operations aren't strictly necessary, this shouldn't cause any
      noticeable difference.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NLi Zefan <lizefan@huawei.com>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      Acked-by: NAristeu Rozanski <aris@redhat.com>
      Acked-by: NMichal Hocko <mhocko@suse.cz>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: Matt Helsley <matthltc@us.ibm.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Balbir Singh <bsingharora@gmail.com>
      bd8815a6
    • T
      cgroup: make hierarchy iterators deal with cgroup_subsys_state instead of cgroup · 492eb21b
      Tejun Heo 提交于
      cgroup is currently in the process of transitioning to using css
      (cgroup_subsys_state) as the primary handle instead of cgroup in
      subsystem API.  For hierarchy iterators, this is beneficial because
      
      * In most cases, css is the only thing subsystems care about anyway.
      
      * On the planned unified hierarchy, iterations for different
        subsystems will need to skip over different subtrees of the
        hierarchy depending on which subsystems are enabled on each cgroup.
        Passing around css makes it unnecessary to explicitly specify the
        subsystem in question as css is intersection between cgroup and
        subsystem
      
      * For the planned unified hierarchy, css's would need to be created
        and destroyed dynamically independent from cgroup hierarchy.  Having
        cgroup core manage css iteration makes enforcing deref rules a lot
        easier.
      
      Most subsystem conversions are straight-forward.  Noteworthy changes
      are
      
      * blkio: cgroup_to_blkcg() is no longer used.  Removed.
      
      * freezer: cgroup_freezer() is no longer used.  Removed.
      
      * devices: cgroup_to_devcgroup() is no longer used.  Removed.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NLi Zefan <lizefan@huawei.com>
      Acked-by: NMichal Hocko <mhocko@suse.cz>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      Acked-by: NAristeu Rozanski <aris@redhat.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Balbir Singh <bsingharora@gmail.com>
      Cc: Matt Helsley <matthltc@us.ibm.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      492eb21b
    • T
      cgroup: pass around cgroup_subsys_state instead of cgroup in file methods · 182446d0
      Tejun Heo 提交于
      cgroup is currently in the process of transitioning to using struct
      cgroup_subsys_state * as the primary handle instead of struct cgroup.
      Please see the previous commit which converts the subsystem methods
      for rationale.
      
      This patch converts all cftype file operations to take @css instead of
      @cgroup.  cftypes for the cgroup core files don't have their subsytem
      pointer set.  These will automatically use the dummy_css added by the
      previous patch and can be converted the same way.
      
      Most subsystem conversions are straight forwards but there are some
      interesting ones.
      
      * freezer: update_if_frozen() is also converted to take @css instead
        of @cgroup for consistency.  This will make the code look simpler
        too once iterators are converted to use css.
      
      * memory/vmpressure: mem_cgroup_from_css() needs to be exported to
        vmpressure while mem_cgroup_from_cont() can be made static.
        Updated accordingly.
      
      * cpu: cgroup_tg() doesn't have any user left.  Removed.
      
      * cpuacct: cgroup_ca() doesn't have any user left.  Removed.
      
      * hugetlb: hugetlb_cgroup_form_cgroup() doesn't have any user left.
        Removed.
      
      * net_cls: cgrp_cls_state() doesn't have any user left.  Removed.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NLi Zefan <lizefan@huawei.com>
      Acked-by: NMichal Hocko <mhocko@suse.cz>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      Acked-by: NAristeu Rozanski <aris@redhat.com>
      Acked-by: NDaniel Wagner <daniel.wagner@bmw-carit.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Balbir Singh <bsingharora@gmail.com>
      Cc: Matt Helsley <matthltc@us.ibm.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      182446d0
  3. 15 5月, 2013 28 次提交
    • T
      blk-throttle: implement proper hierarchy support · 9138125b
      Tejun Heo 提交于
      With the recent updates, blk-throttle is finally ready for proper
      hierarchy support.  Dispatching now honors service_queue->parent_sq
      and propagates correctly.  The only thing missing is setting
      ->parent_sq correctly so that throtl_grp hierarchy matches the cgroup
      hierarchy.
      
      This patch updates throtl_pd_init() such that service_queues form the
      same hierarchy as the cgroup hierarchy if sane_behavior is enabled.
      As this concludes proper hierarchy support for blkcg, the shameful
      .broken_hierarchy tag is removed from blkio_subsys.
      
      v2: Updated blkio-controller.txt as suggested by Vivek.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      Cc: Li Zefan <lizefan@huawei.com>
      9138125b
    • T
      blk-throttle: implement throtl_grp->has_rules[] · 693e751e
      Tejun Heo 提交于
      blk_throtl_bio() has a quick exit path for throtl_grps without limits
      configured.  It looks at the bps and iops limits and if both are not
      configured, the bio is issued immediately.  While this is correct in
      the current flat hierarchy as each throtl_grp behaves completely
      independently, it would become wrong in proper hierarchy mode.  A
      group without any limits could still be limited by one of its
      ancestors and bio's queued for such group should not bypass
      blk-throtl.
      
      As having a quick bypass mechanism is beneficial, this patch
      reimplements the mechanism such that it's correct even with proper
      hierarchy.  throtl_grp->has_rules[] is added.  These booleans are
      updated for the whole subtree whenever a config is updated so that
      has_rules[] of the whole subtree stays synchronized.  They're also
      updated when a new throtl_grp comes online so that it can't escape the
      limits of its ancestors.
      
      As no throtl_grp has another throtl_grp as parent now, this patch
      doesn't yet make any behavior differences.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      693e751e
    • V
      blk-throttle: Account for child group's start time in parent while bio climbs up · 32ee5bc4
      Vivek Goyal 提交于
      With the planned proper hierarchy support, a bio will climb up the
      tree before actually being dispatched. This makes sure bio is also
      subjected to parent's throttling limits, if any.
      
      It might happen that parent is idle and when bio is transferred to
      parent, a new slice starts fresh. But that is incorrect as parents
      wait time should have started when bio was queued in child group and
      causes IOs to be throttled more than configured as they climb the
      hierarchy.
      
      Given the fact that we have not written hierarchical algorithm in a
      way where child's and parents time slices are synchronized, we
      transfer the child's start time to parent if parent was idling.  If
      parent was busy doing dispatch of other bios all this while, this is
      not an issue.
      
      Child's slice start time is passed to parent. Parent looks at its
      last expired slice start time. If child's start time is after parents
      old start time, that means parent had been idle and after parent
      went idle, child had an IO queued. So use child's start time as
      parent start time.
      
      If parent's start time is after child's start time, that means,
      when IO got queued in child group, parent was not idle. But later
      it dispatched some IO, its slice got trimmed and then it went idle.
      After a while child's request got shifted in parent group. In this
      case use parent's old start time as new start time as that's the
      duration of slice we did not use.
      
      This logic is far from perfect as if there are multiple childs
      then first child transferring the bio decides the start time while
      a bio might have queued up even earlier in other child, which is
      yet to be transferred up to parent. In that case we will lose
      time and bandwidth in parent. This patch is just an approximation
      to make situation somewhat better.
      Signed-off-by: NVivek Goyal <vgoyal@redhat.com>
      Signed-off-by: NTejun Heo <tj@kernel.org>
      32ee5bc4
    • T
      blk-throttle: add throtl_qnode for dispatch fairness · c5cc2070
      Tejun Heo 提交于
      With flat hierarchy, there's only single level of dispatching
      happening and fairness beyond that point is the responsibility of the
      rest of the block layer and driver, which usually works out okay;
      however, with the planned hierarchy support,
      service_queue->bio_lists[] can be filled up by bios from a single
      source.  While the limits would still be honored, it'd be very easy to
      starve IOs from siblings or children.
      
      To avoid such starvation, this patch implements throtl_qnode and
      converts service_queue->bio_lists[] to lists of per-source qnodes
      which in turn contains the bio's.  For example, when a bio is
      dispatched from a child group, the bio doesn't get queued on
      ->bio_lists[] directly but it first gets queued on the group's qnode
      which in turn gets queued on service_queue->queued[].  When
      dispatching for the upper level, the ->queued[] list is consumed in
      round-robing order so that the dispatch windows is consumed fairly by
      all IO sources.
      
      There are two ways a bio can come to a throtl_grp - directly queued to
      the group or dispatched from a child.  For the former
      throtl_grp->qnode_on_self[rw] is used.  For the latter, the child's
      ->qnode_on_parent[rw].
      
      Note that this means that the child which is contributing a bio to its
      parent should stay pinned until all its bios are dispatched to its
      grand-parent.  This patch moves blkg refcnting from bio add/remove
      spots to qnode activation/deactivation so that the blkg containing an
      active qnode is always pinned.  As child pins the parent, this is
      sufficient for keeping the relevant sub-tree pinned while bios are in
      flight.
      
      The starvation issue was spotted by Vivek Goyal.
      
      v2: The original patch used the same throtl_grp->qnode_on_self/parent
          for reads and writes causing RWs to be queued incorrectly if there
          already are outstanding IOs in the other direction.  They should
          be throtl_grp->qnode_on_self/parent[2] so that READs and WRITEs
          can use different qnodes.  Spotted by Vivek Goyal.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      c5cc2070
    • T
      blk-throttle: make throtl_pending_timer_fn() ready for hierarchy · 2e48a530
      Tejun Heo 提交于
      throtl_pending_timer_fn() currently assumes that the parent_sq is the
      top level one and the bio's dispatched are ready to be issued;
      however, this assumption will be wrong with proper hierarchy support.
      This patch makes the following changes to make
      throtl_pending_timer_fn() ready for hiearchy.
      
      * If the parent_sq isn't the top-level one, update the parent
        throtl_grp's dispatch time and schedule the next dispatch as
        necessary.  If the parent's dispatch time is now, repeat the
        function for the parent throtl_grp.
      
      * If the parent_sq is the top-level one, kick issue work_item as
        before.
      
      * The debug message printed by throtl_log() now prints out the
        service_queue's nr_queued[] instead of the total nr_queued as the
        latter becomes uninteresting and misleading with hierarchical
        dispatch.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      2e48a530
    • T
      blk-throttle: make tg_dispatch_one_bio() ready for hierarchy · 6bc9c2b4
      Tejun Heo 提交于
      tg_dispatch_one_bio() currently assumes that the parent_sq is the top
      level one and the bio being dispatched is ready to be issued; however,
      this assumption will be wrong with proper hierarchy support.  This
      patch makes the following changes to make tg_dispatch_on_bio() ready
      for hiearchy.
      
      * throtl_data->nr_queued[] is incremented in blk_throtl_bio() instead
        of throtl_add_bio_tg() so that throtl_add_bio_tg() can be used to
        transfer a bio from a child tg to its parent.
      
      * tg_dispatch_one_bio() is updated to distinguish whether its parent
        is another throtl_grp or the throtl_data.  If former, the bio is
        transferred to the parent throtl_grp using throtl_add_bio_tg().  If
        latter, the bio is ready to be issued and put on the top-level
        service_queue's bio_lists[] and throtl_data->nr_queued is
        decremented.
      
      As all throtl_grps currently have the top level service_queue as their
      ->parent_sq, this patch in itself doesn't make any behavior
      difference.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      6bc9c2b4
    • T
      blk-throttle: make blk_throtl_bio() ready for hierarchy · 9e660acf
      Tejun Heo 提交于
      Currently, blk_throtl_bio() issues the passed in bio directly if it's
      within limits of its associated tg (throtl_grp).  This behavior
      becomes incorrect with hierarchy support as the bio should be
      accounted to and throttled by the ancestor throtl_grps too.
      
      This patch makes the direct issue path of blk_throtl_bio() to loop
      until it reaches the top-level service_queue or gets throttled.  If
      the former, the bio can be issued directly; otherwise, it gets queued
      at the first layer it was above limits.
      
      As tg->parent_sq is always the top-level service queue currently, this
      patch in itself doesn't make any behavior differences.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      9e660acf
    • T
      blk-throttle: make blk_throtl_drain() ready for hierarchy · 2a12f0dc
      Tejun Heo 提交于
      The current blk_throtl_drain() assumes that all active throtl_grps are
      queued on throtl_data->service_queue, which won't be true once
      hierarchy support is implemented.
      
      This patch makes blk_throtl_drain() perform post-order walk of the
      blkg hierarchy draining each associated throtl_grp, which guarantees
      that all bios will eventually be pushed to the top-level service_queue
      in throtl_data.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      2a12f0dc
    • T
      blk-throttle: dispatch from throtl_pending_timer_fn() · 6e1a5704
      Tejun Heo 提交于
      Currently, blk_throtl_dispatch_work_fn() is responsible for both
      dispatching bio's from throtl_grp's according to their limits and then
      issuing the dispatched bios.
      
      This patch moves the dispatch part to throtl_pending_timer_fn() so
      that the work item is kicked iff there are bio's to issue.  This is to
      avoid work item execution at each step when hierarchy support is
      enabled.  bio's will be dispatched towards the top-level service_queue
      from the timers at each layer and the work item will only be used to
      issue the bio's which reached the top-level service_queue.
      
      While fetching bio's to issue from bio_lists[],
      blk_throtl_dispatch_work_fn() fetches all READs before WRITEs.  While
      the original code also dispatched READs first, if multiple throtl_grps
      are dispatched on the same run, WRITEs from throtl_grp which is
      dispatched first would precede READs from throtl_grps which are
      dispatched later.  While this is a behavior change, given that the
      previous code already prioritized READs and block layer generally
      prioritizes and segregates READs from WRITEs, this isn't likely to
      make any noticeable differences.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      6e1a5704
    • T
      blk-throttle: implement dispatch looping · 7f52f98c
      Tejun Heo 提交于
      throtl_select_dispatch() only dispatches throtl_quantum bios on each
      invocation.  blk_throtl_dispatch_work_fn() in turn depends on
      throtl_schedule_next_dispatch() scheduling the next dispatch window
      immediately so that undue delays aren't incurred.  This effectively
      chains multiple dispatch work item executions back-to-back when there
      are more than throtl_quantum bios to dispatch on a given tick.
      
      There is no reason to finish the current work item just to repeat it
      immediately.  This patch makes throtl_schedule_next_dispatch() return
      %false without doing anything if the current dispatch window is still
      open and updates blk_throtl_dispatch_work_fn() repeat dispatching
      after cpu_relax() on %false return.
      
      This change will help implementing hierarchy support as dispatching
      will be done from pending_timer and immediate reschedule of timer
      function isn't supported and doesn't make much sense.
      
      While this patch changes how dispatch behaves when there are more than
      throtl_quantum bios to dispatch on a single tick, the behavior change
      is immaterial.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      7f52f98c
    • T
      blk-throttle: separate out throtl_service_queue->pending_timer from throtl_data->dispatch_work · 69df0ab0
      Tejun Heo 提交于
      Currently, throtl_data->dispatch_work is a delayed_work item which
      handles both delayed dispatch and issuing bios.  The two tasks will be
      separated to support proper hierarchy.  To prepare for that, this
      patch separates out the timer into throtl_service_queue->pending_timer
      from throtl_data->dispatch_work and make the latter a work_struct.
      
      * As the timer is now per-service_queue, it's initialized and
        del_sync'd as its corresponding service_queue is created and
        destroyed.  The timer, when triggered, simply schedules
        throtl_data->dispathc_work for execution.
      
      * throtl_schedule_delayed_work() is renamed to
        throtl_schedule_pending_timer() and takes @sq and @expires now.
      
      * Simiarly, throtl_schedule_next_dispatch() now takes @sq, which
        should be the parent_sq of the service_queue which just got a new
        bio or updated.  As the parent_sq is always the top-level
        service_queue now, this doesn't change anything at this point.
      
      This patch doesn't introduce any behavior differences.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      69df0ab0
    • T
      blk-throttle: set REQ_THROTTLED from throtl_charge_bio() and gate stats update with it · 2a0f61e6
      Tejun Heo 提交于
      With proper hierarchy support, a bio can be dispatched multiple times
      until it reaches the top-level service_queue and we don't want to
      update dispatch stats at each step.  They are local stats and will be
      kept local.  If recursive stats are necessary, they should be
      implemented separately and definitely not by updating counters
      recursively on each dispatch.
      
      This patch moves REQ_THROTTLED setting to throtl_charge_bio() and gate
      stats update with it so that dispatch stats are updated only on the
      first time the bio is charged to a throtl_grp, which will always be
      the throtl_grp the bio was originally queued to.
      
      This means that REQ_THROTTLED would be set even for bios which don't
      get throttled.  As we don't want bios to leave blk-throtl with the
      flag set, move REQ_THROTLLED clearing to the end of blk_throtl_bio()
      and clear if the bio is being issued directly.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      2a0f61e6
    • T
      blk-throttle: implement sq_to_tg(), sq_to_td() and throtl_log() · fda6f272
      Tejun Heo 提交于
      Now that both throtl_data and throtl_grp embed throtl_service_queue,
      we can unify throtl_log() and throtl_log_tg().
      
      * sq_to_tg() is added.  This returns the throtl_grp a service_queue is
        embedded in.  If the service_queue is the top-level one embedded in
        throtl_data, NULL is returned.
      
      * sq_to_td() is added.  A service_queue is always associated with a
        throtl_data.  This function finds the associated td and returns it.
      
      * throtl_log() is updated to take throtl_service_queue instead of
        throtl_data.  If the service_queue is one embedded in throtl_grp, it
        prints the same header as throtl_log_tg() did.  If it's one embedded
        in throtl_data, it behaves the same as before.  This renders
        throtl_log_tg() unnecessary.  Removed.
      
      This change is necessary for hierarchy support as we're gonna be using
      the same code paths to dispatch bios to intermediate service_queues
      embedded in throtl_grps and the top-level service_queue embedded in
      throtl_data.
      
      This patch doesn't make any behavior changes.
      
      v2: throtl_log() didn't print a space after blkg path.  Updated so
          that it prints a space after throtl_grp path.  Spotted by Vivek.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      fda6f272
    • T
      blk-throttle: add throtl_service_queue->parent_sq · 77216b04
      Tejun Heo 提交于
      To prepare for hierarchy support, this patch adds
      throtl_service_queue->service_sq which points to the arent
      service_queue.  Currently, for all service_queues embedded in
      throtl_grps, it points to throtl_data->service_queue.  As
      throtl_data->service_queue doesn't have a parent its parent_sq is set
      to NULL.
      
      There are a number of functions which take both throtl_grp *tg and
      throtl_service_queue *parent_sq.  With this patch, the parent
      service_queue can be determined from @tg and the @parent_sq arguments
      are removed.
      
      This patch doesn't make any behavior differences.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      77216b04
    • T
      blk-throttle: generalize update_disptime optimization in blk_throtl_bio() · 0e9f4164
      Tejun Heo 提交于
      When blk_throtl_bio() wants to queue a bio to a tg (throtl_grp), it
      avoids invoking tg_update_disptime() and
      throtl_schedule_next_dispatch() if the tg already has bios queued in
      that direction.  As a new bio is appeneded after the existing ones, it
      can't change the tg's next dispatch time or the parent's dispatch
      schedule.
      
      This optimization is currently open coded in blk_throtl_bio().
      Whether the target biolist was occupied was recorded in a local
      variable and later used to skip disptime update.  This patch moves
      generalizes it so that throtl_add_bio_tg() sets a new flag
      THROTL_TG_WAS_EMPTY if the biolist was empty before the new bio was
      added.  tg_update_disptime() clears the flag automatically.
      blk_throtl_bio() is updated to simply test the flag before updating
      disptime.
      
      This patch doesn't make any functional differences now but will enable
      using the same optimization for recursive dispatch.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      0e9f4164
    • T
      blk-throttle: dispatch to throtl_data->service_queue.bio_lists[] · 651930bc
      Tejun Heo 提交于
      throtl_service_queues will eventually form a tree which is anchored at
      throtl_data->service_queue and queue bios will climb the tree to the
      top service_queue to be executed.
      
      This patch makes the dispatch paths in blk_throtl_dispatch_work_fn()
      and blk_throtl_drain() to dispatch bios to
      throtl_data->service_queue.bio_lists[] instead of the on-stack
      bio_lists.  This will keep the final dispatch to the top level
      service_queue share the same mechanism as dispatches through the rest
      of the hierarchy.
      
      As bio's should be issued in a sleepable context,
      blk_throtl_dispatch_work_fn() transfers all dispatched bio's from the
      service_queue bio_lists[] into an onstack one before dropping
      queue_lock and issuing the bio's.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      651930bc
    • T
      blk-throttle: move bio_lists[] and friends to throtl_service_queue · 73f0d49a
      Tejun Heo 提交于
      throtl_service_queues will eventually form a tree which is anchored at
      throtl_data->service_queue and queue bios will climb the tree to the
      top service_queue to be executed.
      
      This patch moves bio_lists[] and nr_queued[] from throtl_grp to its
      service_queue to prepare for that.  As currently only the
      throtl_data->service_queue is in use, this patch just ends up moving
      throtl_grp->bio_lists[] and ->nr_queued[] to
      throtl_grp->service_queue.bio_lists[] and ->nr_queued[] without making
      any functional differences.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      73f0d49a
    • T
      blk-throttle: add throtl_grp->service_queue · 49a2f1e3
      Tejun Heo 提交于
      Currently, there's single service_queue per queue -
      throtl_data->service_queue.  All active throtl_grp's are queued on the
      queue and dispatched according to their limits.  To support hierarchy,
      this will be expanded such that active throtl_grp's form a tree
      anchored at throtl_data->service_queue and chained through each
      intermediate throtl_grp's service_queue.
      
      This patch adds throtl_grp->service_queue to prepare for hierarchy
      support.  The initialization function - throtl_service_queue_init() -
      is added and replaces the macro initializer.  The newly added
      tg->service_queue isn't used yet.  Following patches will do.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      49a2f1e3
    • T
      blk-throttle: reorganize throtl_service_queue passed around as argument · 0049af73
      Tejun Heo 提交于
      throtl_service_queue will be the building block of hierarchy support
      and will form a tree.  This patch updates its usages as arguments to
      reduce confusion.
      
      * When a service queue is used as the parent role - the host of the
        rbtree - use @parent_sq instead of @sq.
      
      * For functions taking both @tg and @parent_sq, reorder them so that
        the order is (@tg, @parent_sq) not the other way around.  This makes
        the code follow the usual convention of specifying the primary
        target of the operation as the first argument.
      
      This patch doesn't make any functional differences.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      0049af73
    • T
      blk-throttle: pass around throtl_service_queue instead of throtl_data · e2d57e60
      Tejun Heo 提交于
      throtl_service_queue will be used as the basic block to implement
      hierarchy support.  Pass around throtl_service_queue *sq instead of
      throtl_data *td in the following functions which will be used across
      multiple levels of hierarchy.
      
      * [__]throtl_enqueue/dequeue_tg()
      
      * throtl_add_bio_tg()
      
      * tg_update_disptime()
      
      * throtl_select_dispatch()
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      e2d57e60
    • T
      blk-throttle: add backlink pointer from throtl_grp to throtl_data · 0f3457f6
      Tejun Heo 提交于
      Add throtl_grp->td so that the td (throtl_data) a given tg
      (throtl_grp) belongs to can be determined, and remove @td argument
      from functions which take both @td and @tg as the former now can be
      determined from the latter.
      
      This generally simplifies the code and removes a number of cases where
      @td is passed as an argument without being actually used.  This will
      also help hierarchy support implementation.
      
      While at it, in multi-line conditions, move the logical operators
      leading broken lines to the end of the previous line.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      0f3457f6
    • T
      blk-throttle: simplify throtl_grp flag handling · 5b2c16aa
      Tejun Heo 提交于
      blk-throttle is still using function-defining macros to define flag
      handling functions, which went out style at least a decade ago.
      
      Just define the flag as bitmask and use direct bit operations.
      
      This patch doesn't make any functional changes.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      5b2c16aa
    • T
      blk-throttle: rename throtl_rb_root to throtl_service_queue · c9e0332e
      Tejun Heo 提交于
      throtl_rb_root will be expanded to cover more roles for hierarchy
      support.  Rename it to throtl_service_queue and make its fields more
      descriptive.
      
      * rb		-> pending_tree
      * left		-> first_pending
      * count		-> nr_pending
      * min_disptime	-> first_pending_disptime
      
      This patch is purely cosmetic.
      
      Signed-off-by: Tejun Heo <tj@kernel.org
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      c9e0332e
    • T
      blk-throttle: remove pointless throtl_nr_queued() optimizations · 6a525600
      Tejun Heo 提交于
      throtl_nr_queued() is used in several places to avoid performing
      certain operations when the throtl_data is empty.  This usually is
      useless as those paths usually aren't traveled if there's no bio
      queued.
      
      * throtl_schedule_delayed_work() skips scheduling dispatch work item
        if @td doesn't have any bios queued; however, the only case it can
        be called when @td is empty is from tg_set_conf() which isn't
        something we should be optimizing for.
      
      * throtl_schedule_next_dispatch() takes a quick exit if @td is empty;
        however, right after that it triggers BUG if the service tree is
        empty.  The two conditions are equivalent and it can just test
        @st->count for the quick exit.
      
      * blk_throtl_dispatch_work_fn() skips dispatch if @td is empty.  This
        work function isn't usually invoked when @td is empty.  The only
        possibility is from tg_set_conf() and when it happens the normal
        dispatching path can handle empty @td fine.  No need to add special
        skip path.
      
      This patch removes the above three unnecessary optimizations, which
      leave throtl_log() call in blk_throtl_dispatch_work_fn() the only user
      of throtl_nr_queued().  Remove throtl_nr_queued() and open code it in
      throtl_log().  I don't think we need td->nr_queued[] at all.  Maybe we
      can remove it later.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      6a525600
    • T
      blk-throttle: relocate throtl_schedule_delayed_work() · a9131a27
      Tejun Heo 提交于
      Move throtl_schedule_delayed_work() above its first user so that the
      forward declaration can be removed.
      
      This patch is pure relocaiton.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      a9131a27
    • T
      blk-throttle: collapse throtl_dispatch() into the work function · cb76199c
      Tejun Heo 提交于
      blk-throttle is about to go through major restructuring to support
      hierarchy.  Do cosmetic updates in preparation.
      
      * s/throtl_data->throtl_work/throtl_data->dispatch_work/
      
      * s/blk_throtl_work()/blk_throtl_dispatch_work_fn()/
      
      * Collapse throtl_dispatch() into blk_throtl_dispatch_work_fn()
      
      This patch is purely cosmetic.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      cb76199c
    • T
      blk-throttle: remove deferred config application mechanism · 632b4493
      Tejun Heo 提交于
      When bps or iops configuration changes, blk-throttle records the new
      configuration and sets a flag indicating that the config has changed.
      The flag is checked in the bio dispatch path and applied.  This
      deferred config application was necessary due to limitations in blkcg
      framework, which haven't existed for quite a while now.
      
      This patch removes the deferred config application mechanism and
      applies new configurations directly from tg_set_conf(), which is
      simpler.
      
      v2: Dropped unnecessary throtl_schedule_delayed_work() call from
          tg_set_conf() as suggested by Vivek Goyal.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      632b4493
    • T
      blk-throttle: remove spurious throtl_enqueue_tg() call from throtl_select_dispatch() · 2db6314c
      Tejun Heo 提交于
      throtl_select_dispatch() calls throtl_enqueue_tg() right after
      tg_update_disptime(), which always calls the function anyway.  The
      call is, while harmless, unnecessary.  Remove it.
      
      This patch doesn't introduce any behavior difference.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      2db6314c
  4. 06 12月, 2012 1 次提交
    • B
      block: Rename queue dead flag · 3f3299d5
      Bart Van Assche 提交于
      QUEUE_FLAG_DEAD is used to indicate that queuing new requests must
      stop. After this flag has been set queue draining starts. However,
      during the queue draining phase it is still safe to invoke the
      queue's request_fn, so QUEUE_FLAG_DYING is a better name for this
      flag.
      
      This patch has been generated by running the following command
      over the kernel source tree:
      
      git grep -lEw 'blk_queue_dead|QUEUE_FLAG_DEAD' |
          xargs sed -i.tmp -e 's/blk_queue_dead/blk_queue_dying/g'      \
              -e 's/QUEUE_FLAG_DEAD/QUEUE_FLAG_DYING/g';                \
      sed -i.tmp -e "s/QUEUE_FLAG_DYING$(printf \\t)*5/QUEUE_FLAG_DYING$(printf \\t)5/g" \
          include/linux/blkdev.h;                                       \
      sed -i.tmp -e 's/ DEAD/ DYING/g' -e 's/dead queue/a dying queue/' \
          -e 's/Dead queue/A dying queue/' block/blk-core.c
      Signed-off-by: NBart Van Assche <bvanassche@acm.org>
      Acked-by: NTejun Heo <tj@kernel.org>
      Cc: James Bottomley <JBottomley@Parallels.com>
      Cc: Mike Christie <michaelc@cs.wisc.edu>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: Chanho Min <chanho.min@lge.com>
      Signed-off-by: NJens Axboe <axboe@kernel.dk>
      3f3299d5
  5. 22 8月, 2012 1 次提交
    • T
      workqueue: use mod_delayed_work() instead of __cancel + queue · e7c2f967
      Tejun Heo 提交于
      Now that mod_delayed_work() is safe to call from IRQ handlers,
      __cancel_delayed_work() followed by queue_delayed_work() can be
      replaced with mod_delayed_work().
      
      Most conversions are straight-forward except for the following.
      
      * net/core/link_watch.c: linkwatch_schedule_work() was doing a quite
        elaborate dancing around its delayed_work.  Collapse it such that
        linkwatch_work is queued for immediate execution if LW_URGENT and
        existing timer is kept otherwise.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Cc: "David S. Miller" <davem@davemloft.net>
      Cc: Tomi Valkeinen <tomi.valkeinen@ti.com> 
      e7c2f967
  6. 21 8月, 2012 1 次提交
    • T
      workqueue: deprecate system_nrt[_freezable]_wq · 3b07e9ca
      Tejun Heo 提交于
      system_nrt[_freezable]_wq are now spurious.  Mark them deprecated and
      convert all users to system[_freezable]_wq.
      
      If you're cc'd and wondering what's going on: Now all workqueues are
      non-reentrant, so there's no reason to use system_nrt[_freezable]_wq.
      Please use system[_freezable]_wq instead.
      
      This patch doesn't make any functional difference.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-By: NLai Jiangshan <laijs@cn.fujitsu.com>
      
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: David Airlie <airlied@linux.ie>
      Cc: Jiri Kosina <jkosina@suse.cz>
      Cc: "David S. Miller" <davem@davemloft.net>
      Cc: Rusty Russell <rusty@rustcorp.com.au>
      Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
      Cc: David Howells <dhowells@redhat.com>
      3b07e9ca
  7. 25 6月, 2012 1 次提交
    • T
      block: allocate io_context upfront · 7f4b35d1
      Tejun Heo 提交于
      Block layer very lazy allocation of ioc.  It waits until the moment
      ioc is absolutely necessary; unfortunately, that time could be inside
      queue lock and __get_request() performs unlock - try alloc - retry
      dancing.
      
      Just allocate it up-front on entry to block layer.  We're not saving
      the rain forest by deferring it to the last possible moment and
      complicating things unnecessarily.
      
      This patch is to prepare for further updates to request allocation
      path.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Acked-by: NVivek Goyal <vgoyal@redhat.com>
      Signed-off-by: NJens Axboe <axboe@kernel.dk>
      7f4b35d1
  8. 23 5月, 2012 1 次提交
  9. 20 4月, 2012 3 次提交
    • T
      blkcg: collapse blkcg_policy_ops into blkcg_policy · f9fcc2d3
      Tejun Heo 提交于
      There's no reason to keep blkcg_policy_ops separate.  Collapse it into
      blkcg_policy.
      
      This patch doesn't introduce any functional change.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Cc: Vivek Goyal <vgoyal@redhat.com>
      Signed-off-by: NJens Axboe <axboe@kernel.dk>
      f9fcc2d3
    • T
      blkcg: embed struct blkg_policy_data in policy specific data · f95a04af
      Tejun Heo 提交于
      Currently blkg_policy_data carries policy specific data as char flex
      array instead of being embedded in policy specific data.  This was
      forced by oddities around blkg allocation which are all gone now.
      
      This patch makes blkg_policy_data embedded in policy specific data -
      throtl_grp and cfq_group so that it's more conventional and consistent
      with how io_cq is handled.
      
      * blkcg_policy->pdata_size is renamed to ->pd_size.
      
      * Functions which used to take void *pdata now takes struct
        blkg_policy_data *pd.
      
      * blkg_to_pdata/pdata_to_blkg() updated to blkg_to_pd/pd_to_blkg().
      
      * Dummy struct blkg_policy_data definition added.  Dummy
        pdata_to_blkg() definition was unused and inconsistent with the
        non-dummy version - correct dummy pd_to_blkg() added.
      
      * throtl and cfq updated accordingly.
      
      * As dummy blkg_to_pd/pd_to_blkg() are provided,
        blkg_to_cfqg/cfqg_to_blkg() don't need to be ifdef'd.  Moved outside
        ifdef block.
      
      This patch doesn't introduce any functional change.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Cc: Vivek Goyal <vgoyal@redhat.com>
      Signed-off-by: NJens Axboe <axboe@kernel.dk>
      f95a04af
    • T
      blkcg: mass rename of blkcg API · 3c798398
      Tejun Heo 提交于
      During the recent blkcg cleanup, most of blkcg API has changed to such
      extent that mass renaming wouldn't cause any noticeable pain.  Take
      the chance and cleanup the naming.
      
      * Rename blkio_cgroup to blkcg.
      
      * Drop blkio / blkiocg prefixes and consistently use blkcg.
      
      * Rename blkio_group to blkcg_gq, which is consistent with io_cq but
        keep the blkg prefix / variable name.
      
      * Rename policy method type and field names to signify they're dealing
        with policy data.
      
      * Rename blkio_policy_type to blkcg_policy.
      
      This patch doesn't cause any functional change.
      Signed-off-by: NTejun Heo <tj@kernel.org>
      Cc: Vivek Goyal <vgoyal@redhat.com>
      Signed-off-by: NJens Axboe <axboe@kernel.dk>
      3c798398