[PATCH] sched/fair: Avoid calculating curr's key twice in pick_eevdf()

Currently, pick_eevdf() calls entity_eligible() with cfs_rq and curr that
calls vruntime_eligible() and gives the curr->vruntime as the parameter to
vruntime_eligible().

pick_eevdf() checks if curr exists and is on rq before calling
entity_eligible().

That means when entity_eligible() is called, the first 2 checks are not
needed since we already check them in pick_eevdf().

And the key value is calculated by the given vruntime (That is,
in this call path is curr's vruntime) minus zero vruntime.

But above, we do the same calculation with entity_key(),
that does the same calculation with the given entity's
vruntime. But the given entity is curr.

And our parameter vruntime is curr's too. We do the same calculation
twice.

Add curr_eligible that skips the curr exists and on rq checks
and calculates the key once. No functional change intended.

Signed-off-by: Kayra Cizmeci <[email protected]>
---
I boot tested the changes on x86_64 and checked if the behavior
would be the same with checking the results with WARN_ON_ONCE.

I also ran some tests with perf stat, but the results we're noisy,
so I thought that making assumptions with them would be wrong.

 kernel/sched/fair.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..2ec3040bc479 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -936,6 +936,33 @@ static int vruntime_eligible(struct cfs_rq *cfs_rq, u64 vruntime)
 #endif
 }
 
+static int curr_eligible(struct cfs_rq *cfs_rq)
+{
+	struct sched_entity *curr = cfs_rq->curr;
+	s64 key, avg = cfs_rq->sum_w_vruntime;
+	long load = cfs_rq->sum_weight;
+	unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
+
+	key = entity_key(cfs_rq, curr);
+	avg += key * weight;
+	load += weight;
+
+#ifdef CONFIG_64BIT
+#ifdef CONFIG_ARCH_SUPPORTS_INT128
+	return avg >= (__int128)key * load;
+#else
+	s64 rhs;
+
+	if (check_mul_overflow(key, load, &rhs))
+		return key <= 0;
+
+	return avg >= rhs;
+#endif
+#else
+	return avg >= key * load;
+#endif
+}
+
 int entity_eligible(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	return vruntime_eligible(cfs_rq, se->vruntime);
@@ -1157,7 +1184,7 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq, bool protect)
 		return cfs_rq->next;
 	}
 
-	if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr)))
+	if (curr && (!curr->on_rq || !curr_eligible(cfs_rq)))
 		curr = NULL;
 
 	if (curr && protect && protect_slice(curr))
-- 
2.53.0