[PATCH 08/23] rteval: mcp-server: Add query and filter capabilities
John Kacur <[email protected]>
| Newsgroups | org.kernel.vger.linux-rt-users |
|---|---|
| Message-ID | <[email protected]> |
Add three new tools to the MCP server for advanced result analysis: filter_results - Filter rteval results by multiple criteria including kernel version pattern, RT kernel status, date range, minimum test duration, and maximum latency threshold. Enables targeted analysis of specific test configurations. find_best_worst - Identify optimal and poorest performing runs based on configurable latency metrics (maximum, mean, or median). Returns ranked results with kernel version, date, and latency information. compare_to_baseline - Regression detection by comparing multiple results against a baseline file. Detects and reports regressions above a configurable threshold percentage (default 10%), showing detailed percentage changes for all metrics. Updated README with tool descriptions, usage examples, and detailed feature documentation to help users understand the new capabilities. Assisted-by: Claude:claude-sonnet-4-5 Signed-off-by: John Kacur <[email protected]> --- mcp-server/README.md | 81 +++++++- mcp-server/server.py | 450 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 530 insertions(+), 1 deletion(-) diff --git a/mcp-server/README.md b/mcp-server/README.md index 1e09bd63775c..7ce028b512f5 100644 --- a/mcp-server/README.md +++ b/mcp-server/README.md @@ -33,9 +33,18 @@ sudo dnf install python3-mcp python3-mcp+cli python3-lxml ## Available Tools +### Basic Analysis - **list_results**: List rteval result files in a directory - **parse_result**: Parse an rteval XML result file and extract key metrics - **compare_results**: Compare two rteval result files side-by-side +- **batch_analysis**: Analyze multiple rteval result files and aggregate statistics + +### Query & Filter +- **filter_results**: Filter rteval results by kernel version, date range, test duration, and latency thresholds +- **find_best_worst**: Find best and worst rteval runs based on latency metrics (maximum, mean, or median) +- **compare_to_baseline**: Compare multiple results against a baseline file and detect regressions + +### Log Access - **list_logs**: List available log files in an rteval result directory - **read_log**: Read log content with optional filtering (head/tail/grep) @@ -56,9 +65,40 @@ The server is configured as a Claude Code plugin. After installation: 3. Restart Claude Code to load the plugin 4. Tools will be available with the prefix: `mcp__plugin_rteval-mcp_rteval__` -Example: +Example queries: ``` list rteval results in ~/src/rteval +find the best and worst rteval runs +show me rteval results with kernel 7.0 +compare rteval results to a baseline +``` + +### Query & Filter Examples + +**Find best and worst runs:** +``` +Find the 3 best and worst rteval runs by maximum latency +Which rteval runs had the lowest mean latency? +``` + +**Filter by criteria:** +``` +Show me rteval results with kernel version 7.0 +Find rteval results from June 2026 +Show results with max latency under 3500 microseconds +Filter rteval results by RT kernel +``` + +**Baseline comparison:** +``` +Compare all rteval results to rteval-20260605-1/summary.xml +Check for regressions against the baseline with 5% threshold +``` + +**Batch analysis:** +``` +Analyze all rteval results and show aggregate statistics +What's the average max latency across all runs? ``` ### Using with MCP Inspector @@ -67,6 +107,45 @@ list rteval results in ~/src/rteval mcp dev server.py ``` +## Features in Detail + +### filter_results +Filter rteval results by multiple criteria: +- **kernel_pattern**: Match kernel version (e.g., "7.0", "rt", "6.17") +- **is_rt**: Filter by RT kernel (true/false) +- **date_from/date_to**: Date range filter (YYYY-MM-DD format) +- **min_duration_minutes**: Minimum test duration +- **max_latency_threshold**: Maximum acceptable latency in µs + +Example: Find all RT kernel runs from June 2026 with max latency under 4000 µs + +### find_best_worst +Identify optimal and poorest performing runs: +- **metric**: Choose maximum, mean, or median latency +- **count**: Number of best/worst results to return (default: 5) + +Shows ranked results with kernel version, date, and latency metrics. + +### compare_to_baseline +Regression detection against a baseline result: +- **baseline_file**: Reference rteval XML file +- **directory**: Directory containing results to compare +- **threshold_percent**: Alert threshold for regressions (default: 10%) + +Reports: +- Regressions (results exceeding threshold) +- Passes (results within threshold) +- Detailed percentage changes for all metrics + +### batch_analysis +Aggregate statistics across multiple runs: +- Total files analyzed +- Date range coverage +- Min/max/average latencies across all runs +- Individual result summaries + +Useful for understanding performance trends over time. + ## Development This is a work in progress. The MCP server is being developed in a branch of the diff --git a/mcp-server/server.py b/mcp-server/server.py index aa8cfe1102d9..8dbcd9c62fd1 100755 --- a/mcp-server/server.py +++ b/mcp-server/server.py @@ -297,6 +297,91 @@ async def list_tools() -> list[Tool]: "required": [], }, ), + Tool( + name="filter_results", + description="Filter rteval results by kernel version, date range, and test parameters", + inputSchema={ + "type": "object", + "properties": { + "directory": { + "type": "string", + "description": "Directory to search (default: current directory)", + }, + "kernel_pattern": { + "type": "string", + "description": "Filter by kernel version pattern (e.g., '6.17', 'rt', '7.0')", + }, + "is_rt": { + "type": "boolean", + "description": "Filter by RT kernel (true/false)", + }, + "date_from": { + "type": "string", + "description": "Start date (YYYY-MM-DD)", + }, + "date_to": { + "type": "string", + "description": "End date (YYYY-MM-DD)", + }, + "min_duration_minutes": { + "type": "integer", + "description": "Minimum test duration in minutes", + }, + "max_latency_threshold": { + "type": "number", + "description": "Filter results with max latency below this threshold (µs)", + }, + }, + "required": [], + }, + ), + Tool( + name="find_best_worst", + description="Find best and worst rteval runs based on latency metrics", + inputSchema={ + "type": "object", + "properties": { + "directory": { + "type": "string", + "description": "Directory to search (default: current directory)", + }, + "metric": { + "type": "string", + "description": "Metric to evaluate: 'maximum', 'mean', or 'median' (default: maximum)", + "default": "maximum", + }, + "count": { + "type": "integer", + "description": "Number of best/worst results to return (default: 5)", + "default": 5, + }, + }, + "required": [], + }, + ), + Tool( + name="compare_to_baseline", + description="Compare multiple rteval results against a baseline result", + inputSchema={ + "type": "object", + "properties": { + "baseline_file": { + "type": "string", + "description": "Path to baseline rteval XML file", + }, + "directory": { + "type": "string", + "description": "Directory containing results to compare (default: current directory)", + }, + "threshold_percent": { + "type": "number", + "description": "Alert if results exceed baseline by this percentage (default: 10)", + "default": 10, + }, + }, + "required": ["baseline_file"], + }, + ), ] @@ -889,6 +974,371 @@ async def call_tool(name: str, arguments: Any) -> list[TextContent]: text=f"Error in batch analysis: {str(e)}" )] + elif name == "filter_results": + directory = arguments.get("directory", ".") + kernel_pattern = arguments.get("kernel_pattern") + is_rt = arguments.get("is_rt") + date_from = arguments.get("date_from") + date_to = arguments.get("date_to") + min_duration = arguments.get("min_duration_minutes") + max_threshold = arguments.get("max_latency_threshold") + + try: + path = Path(directory) + if not path.exists(): + return [TextContent( + type="text", + text=f"Error: Directory '{directory}' does not exist" + )] + + # Find all XML files + files = list(path.glob("*.xml")) + if not files: + return [TextContent( + type="text", + text=f"No XML files found in '{directory}'" + )] + + # Parse and filter + filtered = [] + for file_path in files: + try: + data = extract_rteval_data(str(file_path)) + + # Apply filters + if kernel_pattern and kernel_pattern.lower() not in data["system_info"].get("kernel", "").lower(): + continue + + if is_rt is not None and data["system_info"].get("is_RT") != is_rt: + continue + + # Date filtering + if date_from or date_to: + run_date = data["run_info"].get("date") + if run_date: + if date_from and run_date < date_from: + continue + if date_to and run_date > date_to: + continue + + # Duration filtering + if min_duration: + ri = data["run_info"] + total_minutes = (ri.get("days", 0) * 24 * 60 + + ri.get("hours", 0) * 60 + + ri.get("minutes", 0)) + if total_minutes < min_duration: + continue + + # Max latency threshold + if max_threshold: + max_lat = None + for mtype in ["timerlat", "cyclictest"]: + if mtype in data["measurements"]: + max_val = data["measurements"][mtype].get("maximum", {}).get("value") + if max_val: + try: + max_lat = float(max_val) + break + except ValueError: + pass + + if max_lat is None or max_lat > max_threshold: + continue + + filtered.append(data) + + except Exception: + continue + + if not filtered: + return [TextContent( + type="text", + text="No results matched the filter criteria" + )] + + # Build output + result = f"Filtered Results ({len(filtered)} of {len(files)} files matched):\n" + result += "=" * 60 + "\n\n" + + for data in filtered: + result += f"{Path(data['file']).name}:\n" + if "date" in data["run_info"]: + result += f" Date: {data['run_info']['date']}\n" + if "kernel" in data["system_info"]: + result += f" Kernel: {data['system_info']['kernel']}" + if "is_RT" in data["system_info"]: + result += f" (RT: {'Yes' if data['system_info']['is_RT'] else 'No'})" + result += "\n" + + # Show key metrics + for mtype in ["timerlat", "cyclictest"]: + if mtype in data["measurements"]: + meas = data["measurements"][mtype] + if "maximum" in meas: + result += f" Max latency: {meas['maximum']['value']} {meas['maximum']['unit']}\n" + if "mean" in meas: + result += f" Mean latency: {meas['mean']['value']} {meas['mean']['unit']}\n" + break + + result += "\n" + + return [TextContent(type="text", text=result)] + + except Exception as e: + return [TextContent( + type="text", + text=f"Error filtering results: {str(e)}" + )] + + elif name == "find_best_worst": + directory = arguments.get("directory", ".") + metric = arguments.get("metric", "maximum") + count = arguments.get("count", 5) + + try: + path = Path(directory) + if not path.exists(): + return [TextContent( + type="text", + text=f"Error: Directory '{directory}' does not exist" + )] + + # Find all XML files + files = list(path.glob("*.xml")) + if not files: + return [TextContent( + type="text", + text=f"No XML files found in '{directory}'" + )] + + # Parse and collect metrics + results_with_metrics = [] + for file_path in files: + try: + data = extract_rteval_data(str(file_path)) + + # Extract the requested metric + metric_value = None + for mtype in ["timerlat", "cyclictest"]: + if mtype in data["measurements"]: + metric_data = data["measurements"][mtype].get(metric) + if metric_data: + try: + metric_value = float(metric_data["value"]) + break + except ValueError: + pass + + if metric_value is not None: + results_with_metrics.append((data, metric_value)) + + except Exception: + continue + + if not results_with_metrics: + return [TextContent( + type="text", + text=f"No results found with {metric} metric" + )] + + # Sort by metric value + results_with_metrics.sort(key=lambda x: x[1]) + + # Build output + result = f"Best and Worst Results (by {metric} latency):\n" + result += "=" * 60 + "\n\n" + + # Best results (lowest latency) + result += f"BEST {min(count, len(results_with_metrics))} Results (lowest {metric}):\n" + result += "-" * 60 + "\n" + for i, (data, metric_val) in enumerate(results_with_metrics[:count], 1): + result += f"\n{i}. {Path(data['file']).name}\n" + result += f" {metric.capitalize()}: {metric_val:.2f} µs\n" + if "kernel" in data["system_info"]: + result += f" Kernel: {data['system_info']['kernel']}\n" + if "date" in data["run_info"]: + result += f" Date: {data['run_info']['date']}\n" + + # Worst results (highest latency) + result += f"\nWORST {min(count, len(results_with_metrics))} Results (highest {metric}):\n" + result += "-" * 60 + "\n" + for i, (data, metric_val) in enumerate(reversed(results_with_metrics[-count:]), 1): + result += f"\n{i}. {Path(data['file']).name}\n" + result += f" {metric.capitalize()}: {metric_val:.2f} µs\n" + if "kernel" in data["system_info"]: + result += f" Kernel: {data['system_info']['kernel']}\n" + if "date" in data["run_info"]: + result += f" Date: {data['run_info']['date']}\n" + + return [TextContent(type="text", text=result)] + + except Exception as e: + return [TextContent( + type="text", + text=f"Error finding best/worst results: {str(e)}" + )] + + elif name == "compare_to_baseline": + baseline_file = arguments["baseline_file"] + directory = arguments.get("directory", ".") + threshold_percent = arguments.get("threshold_percent", 10) + + try: + # Load baseline + if not Path(baseline_file).exists(): + return [TextContent( + type="text", + text=f"Error: Baseline file '{baseline_file}' does not exist" + )] + + baseline_data = extract_rteval_data(baseline_file) + + # Get baseline metrics + baseline_metrics = {} + for mtype in ["timerlat", "cyclictest"]: + if mtype in baseline_data["measurements"]: + for metric in ["maximum", "mean", "median"]: + if metric in baseline_data["measurements"][mtype]: + try: + baseline_metrics[metric] = float( + baseline_data["measurements"][mtype][metric]["value"] + ) + except ValueError: + pass + break + + if not baseline_metrics: + return [TextContent( + type="text", + text="Could not extract metrics from baseline file" + )] + + # Find files to compare + path = Path(directory) + if not path.exists(): + return [TextContent( + type="text", + text=f"Error: Directory '{directory}' does not exist" + )] + + files = list(path.glob("*.xml")) + if not files: + return [TextContent( + type="text", + text=f"No XML files found in '{directory}'" + )] + + # Build comparison report + result = f"Baseline Comparison Report\n" + result += "=" * 60 + "\n\n" + result += f"Baseline: {Path(baseline_file).name}\n" + result += f"Baseline Metrics:\n" + for metric, value in baseline_metrics.items(): + result += f" {metric}: {value:.2f} µs\n" + result += f"\nThreshold: {threshold_percent}% above baseline\n" + result += "=" * 60 + "\n\n" + + regressions = [] + passes = [] + + for file_path in files: + # Skip baseline file itself + if str(file_path) == str(Path(baseline_file).resolve()): + continue + + try: + data = extract_rteval_data(str(file_path)) + + # Extract metrics + test_metrics = {} + for mtype in ["timerlat", "cyclictest"]: + if mtype in data["measurements"]: + for metric in ["maximum", "mean", "median"]: + if metric in data["measurements"][mtype]: + try: + test_metrics[metric] = float( + data["measurements"][mtype][metric]["value"] + ) + except ValueError: + pass + break + + if not test_metrics: + continue + + # Check for regressions + has_regression = False + comparison = { + "file": Path(file_path).name, + "kernel": data["system_info"].get("kernel", "unknown"), + "date": data["run_info"].get("date", "unknown"), + "metrics": {} + } + + for metric in baseline_metrics: + if metric in test_metrics: + baseline_val = baseline_metrics[metric] + test_val = test_metrics[metric] + diff = test_val - baseline_val + pct_change = (diff / baseline_val) * 100 if baseline_val != 0 else 0 + + comparison["metrics"][metric] = { + "baseline": baseline_val, + "test": test_val, + "diff": diff, + "pct_change": pct_change + } + + if pct_change > threshold_percent: + has_regression = True + + if has_regression: + regressions.append(comparison) + else: + passes.append(comparison) + + except Exception: + continue + + # Report regressions + if regressions: + result += f"REGRESSIONS DETECTED ({len(regressions)} files):\n" + result += "-" * 60 + "\n\n" + for comp in regressions: + result += f"{comp['file']}:\n" + result += f" Kernel: {comp['kernel']}\n" + result += f" Date: {comp['date']}\n" + for metric, values in comp["metrics"].items(): + result += f" {metric}:\n" + result += f" Baseline: {values['baseline']:.2f} µs\n" + result += f" Test: {values['test']:.2f} µs\n" + result += f" Change: {values['pct_change']:+.2f}%" + if values['pct_change'] > threshold_percent: + result += " ⚠️ REGRESSION" + result += "\n" + result += "\n" + + # Report passes + if passes: + result += f"\nPASSED COMPARISON ({len(passes)} files):\n" + result += "-" * 60 + "\n\n" + for comp in passes: + result += f"{comp['file']}: " + max_pct = max(m['pct_change'] for m in comp['metrics'].values()) + result += f"Max change: {max_pct:+.2f}% ✓\n" + + if not regressions and not passes: + result += "No comparable results found\n" + + return [TextContent(type="text", text=result)] + + except Exception as e: + return [TextContent( + type="text", + text=f"Error comparing to baseline: {str(e)}" + )] + else: return [TextContent( type="text", -- 2.55.0