sched/core: Fix DEBUG_SPINLOCK annotation for rq->lock

Web:        https://git.kernel.org/torvalds/c/269d599271fa604f09d5cb0093c5dd5d59964dd5
Commit:     269d599271fa604f09d5cb0093c5dd5d59964dd5
Parent:     a7711602c7b79950ea437178f601b52ab08ef659
Refname:    refs/heads/master
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Tue Feb 6 17:52:13 2018 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue Feb 13 11:44:41 2018 +0100

    sched/core: Fix DEBUG_SPINLOCK annotation for rq->lock
    
    Mark noticed that he had sporadic "spinlock recursion" warnings from
    the DEBUG_SPINLOCK code. Now rq->lock is special in that the owner
    changes in the middle of a context switch.
    
    It so happens that we fix up the lock.owner too late, @prev can run
    (remotely) the moment prev->on_cpu is cleared, this then allows @prev
    to again try and acquire this rq->lock and trigger this warning.
    
    So we have to switch lock.owner before clearing prev->on_cpu.
    
    Do this by moving the DEBUG_SPINLOCK annotation from after switch_to()
    to before switch_to() and collect all lockdep annotations there into
    prepare_lock_switch() to mirror the existing finish_lock_switch().
    
    Debugged-by: Mark Rutland <[email protected]>
    Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
    Acked-by: Mark Rutland <[email protected]>
    Cc: Linus Torvalds <[email protected]>
    Cc: Peter Zijlstra <[email protected]>
    Cc: Thomas Gleixner <[email protected]>
    Signed-off-by: Ingo Molnar <[email protected]>
---
 kernel/sched/core.c | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index bf724c1952ea..e7c535eee0a6 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2601,19 +2601,31 @@ static inline void finish_task(struct task_struct *prev)
 #endif
 }
 
-static inline void finish_lock_switch(struct rq *rq)
+static inline void
+prepare_lock_switch(struct rq *rq, struct task_struct *next, struct rq_flags *rf)
 {
+	/*
+	 * Since the runqueue lock will be released by the next
+	 * task (which is an invalid locking op but in the case
+	 * of the scheduler it's an obvious special-case), so we
+	 * do an early lockdep release here:
+	 */
+	rq_unpin_lock(rq, rf);
+	spin_release(&rq->lock.dep_map, 1, _THIS_IP_);
 #ifdef CONFIG_DEBUG_SPINLOCK
 	/* this is a valid case when another task releases the spinlock */
-	rq->lock.owner = current;
+	rq->lock.owner = next;
 #endif
+}
+
+static inline void finish_lock_switch(struct rq *rq)
+{
 	/*
 	 * If we are tracking spinlock dependencies then we have to
 	 * fix up the runqueue lock - which gets 'carried over' from
 	 * prev into current:
 	 */
 	spin_acquire(&rq->lock.dep_map, 0, 0, _THIS_IP_);