P2-08: Matrix evaluator makes an LLM call — defeats purpose of "fast local path" #18

New issue

Closed

opened 2026-06-16 13:57:02 +00:00 by Artur · 0 comments

Artur commented

2026-06-16 13:57:02 +00:00

Owner

Severity: P2 (Medium)
File: decider/matrix.py

Problem

MatrixEvaluator.evaluate() calls call_llm() to score each principle:

prompt = self._build_prompt(situation_safe, context_safe)
raw = call_llm(prompt, self.llm_config)  # ← LLM call
scores = self._parse_scores(raw)

This means:

Matrix mode = 1 LLM call (same as llm mode)
The matrix weighting is applied on top of LLM scores — redundant computation
The claim that matrix is "faster" than ensemble is false (both make LLM calls, matrix just makes fewer)
A truly fast local path would classify situations WITHOUT any LLM call

Fix

Implement a rule-based classifier that maps situation keywords/patterns to scores:

Parse situation for risk indicators: "rm -rf", "DROP TABLE", "DELETE FROM", "format", "reboot" → lower scores
Parse for safe indicators: "install", "update", "config", "read", "list" → higher scores
Weight by tool name (terminal=lower, read_file=higher)
Apply the decision matrix (fully rule-based, zero LLM calls)

This gives matrix mode a genuine speed advantage: ~1ms vs ~5s for LLM-based modes. The LLM should be used only when the rule-based classifier returns "ambiguous" (middle confidence band).

**Severity**: P2 (Medium) **File**: `decider/matrix.py` ## Problem `MatrixEvaluator.evaluate()` calls `call_llm()` to score each principle: ```python prompt = self._build_prompt(situation_safe, context_safe) raw = call_llm(prompt, self.llm_config) # ← LLM call scores = self._parse_scores(raw) ``` This means: - Matrix mode = 1 LLM call (same as llm mode) - The matrix weighting is applied on top of LLM scores — redundant computation - The claim that matrix is "faster" than ensemble is false (both make LLM calls, matrix just makes fewer) - A truly fast local path would classify situations WITHOUT any LLM call ## Fix Implement a rule-based classifier that maps situation keywords/patterns to scores: 1. Parse situation for risk indicators: "rm -rf", "DROP TABLE", "DELETE FROM", "format", "reboot" → lower scores 2. Parse for safe indicators: "install", "update", "config", "read", "list" → higher scores 3. Weight by tool name (terminal=lower, read_file=higher) 4. Apply the decision matrix (fully rule-based, zero LLM calls) This gives matrix mode a genuine speed advantage: **~1ms vs ~5s** for LLM-based modes. The LLM should be used only when the rule-based classifier returns "ambiguous" (middle confidence band).

Artur added the

enhancement

reliability

P2-Medium

labels

2026-06-16 13:57:02 +00:00

Artur referenced this issue from a commit

2026-06-16 14:18:56 +00:00

feat: P2-08 — rule-based matrix evaluator (no LLM, ~1ms fast path)

Artur closed this issue

2026-06-16 14:18:56 +00:00

Artur referenced this issue from a commit