[layerindex-web][PATCH 1/5] layerindex/views.py: fix StatsView statistics page timeout
Tim Orling <[email protected]>
| Newsgroups | org.yoctoproject.lists.yocto-patches |
|---|---|
| Message-ID | <[email protected]> |
The per-branch statistics query used a single ORM queryset with five Count(..., distinct=True) annotations spanning deep reverse FK chains (layerbranch__recipe, layerbranch__bbclass, etc.). Django translates this into one SQL query with multiple LEFT OUTER JOINs across all those tables simultaneously. At production data volumes (~541 layers, ~25k recipes) the resulting query exhausts gunicorn worker memory/time, causing a 504 Gateway Time-out. Fix by iterating over each visible branch individually and issuing simple COUNT queries per table. This replaces one huge cross-join with N×5 small indexed WHERE-clause queries that are far cheaper and independently cacheable. The template is unchanged; Django's template engine resolves dict keys and object attributes identically with dot notation. [YOCTO #15391] AI-Generated: Claude Cowork Sonnet 4.6 Signed-off-by: Tim Orling <[email protected]> --- layerindex/views.py | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/layerindex/views.py b/layerindex/views.py index 3cf91d2..23bafd0 100644 --- a/layerindex/views.py +++ b/layerindex/views.py @@ -1522,12 +1522,23 @@ class StatsView(TemplateView): context['class_count_distinct'] = BBClass.objects.values('name').distinct().count() context['machine_count_distinct'] = Machine.objects.values('name').distinct().count() context['distro_count_distinct'] = Distro.objects.values('name').distinct().count() - context['perbranch'] = Branch.objects.filter(hidden=False).order_by('sort_priority').annotate( - layer_count=Count('layerbranch', distinct=True), - recipe_count=Count('layerbranch__recipe', distinct=True), - class_count=Count('layerbranch__bbclass', distinct=True), - machine_count=Count('layerbranch__machine', distinct=True), - distro_count=Count('layerbranch__distro', distinct=True)) + # Compute per-branch counts with individual queries rather than a single + # annotated queryset. The multi-Count(distinct=True) approach generates one + # large SQL query with many LEFT JOINs that causes a gunicorn worker timeout + # at production data volumes. Simple per-branch COUNT queries are far cheaper. + # See: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15391 + perbranch = [] + for branch in Branch.objects.filter(hidden=False).order_by('sort_priority'): + perbranch.append({ + 'name': branch.name, + 'updates_enabled': branch.updates_enabled, + 'layer_count': LayerBranch.objects.filter(branch=branch).count(), + 'recipe_count': Recipe.objects.filter(layerbranch__branch=branch).count(), + 'class_count': BBClass.objects.filter(layerbranch__branch=branch).count(), + 'machine_count': Machine.objects.filter(layerbranch__branch=branch).count(), + 'distro_count': Distro.objects.filter(layerbranch__branch=branch).count(), + }) + context['perbranch'] = perbranch return context -- 2.54.0