fix(ci): add post-generation evidence verification to pr-review
Some checks failed
Test / rust-fmt-check (pull_request) Successful in 1m42s
Test / frontend-typecheck (pull_request) Successful in 1m42s
Test / frontend-tests (pull_request) Successful in 1m42s
Test / rust-clippy (pull_request) Successful in 3m16s
PR Review Automation / review (pull_request) Failing after 4m33s
Test / rust-tests (pull_request) Successful in 4m54s

qwen3-coder-next fabricates plausible-looking code in its Evidence
blocks instead of quoting from the actual files provided. This adds a
Python verification step that greps each fenced code block against the
real changed files and tags any finding whose evidence cannot be found
as UNVERIFIED.

This is a safeguard, not a fix — the model is fundamentally unreliable
for grounded code review. The longer-term fix is to replace qwen3-coder
with a model that stays grounded to context (Claude Haiku, devstral,
or deepseek-coder-v2 via the LiteLLM proxy / vLLM at 172.0.1.42).
This commit is contained in:
Shaun Arman 2026-05-31 14:41:47 -05:00
parent 6373f0b09c
commit cf5bc83b75

View File

@ -144,6 +144,69 @@ jobs:
echo "Review length: ${#REVIEW} chars"
echo "$REVIEW" > /tmp/pr_review.txt
- name: Verify findings against codebase
if: steps.analyze.outcome == 'success'
shell: bash
run: |
set -euo pipefail
# For each finding that contains a fenced code block under "Evidence:",
# grep at least one substantial line of that block against the actual changed
# files. If nothing matches, prepend a visible UNVERIFIED tag so reviewers
# know the model fabricated the evidence.
python3 - << 'PYEOF'
import re, os
review = open('/tmp/pr_review.txt').read()
filelist = [f.strip() for f in open('/tmp/pr_files.txt') if f.strip()]
# Load content of every changed file
repo_text = {}
for path in filelist:
if os.path.isfile(path):
try:
repo_text[path] = open(path).read()
except Exception:
pass
all_content = '\n'.join(repo_text.values())
def evidence_exists(block: str) -> bool:
"""True if ≥1 significant line from the block is found verbatim in changed files."""
for raw in block.splitlines():
line = raw.lstrip('+-').strip()
# Skip blank, very short, pure-comment, or diff-header lines
if len(line) < 20:
continue
if line.startswith(('//','#','/*','*','Fix:','Evidence:','---','+++')):
continue
if line in all_content:
return True
return False
# Split on finding markers; re-join after optional tagging
severity_re = re.compile(r'\[(BLOCKER|WARNING|SUGGESTION)\]')
def tag_if_unverified(finding_text: str) -> str:
code_match = re.search(r'```[^\n]*\n(.*?)```', finding_text, re.DOTALL)
if code_match and not evidence_exists(code_match.group(1)):
# Replace first severity tag with a prefixed version
return severity_re.sub(
lambda m: f'[{m.group(1)} — ⚠️ UNVERIFIED: evidence not found in PR files]',
finding_text, count=1
)
return finding_text
# Split review into preamble + individual finding blocks
# Each block starts at a severity marker line
parts = re.split(r'(?=^\[(?:BLOCKER|WARNING|SUGGESTION)\])', review, flags=re.MULTILINE)
result = parts[0] # preamble (Summary, etc.)
for block in parts[1:]:
result += tag_if_unverified(block)
open('/tmp/pr_review.txt', 'w').write(result)
print(f"Verification complete — {len(parts)-1} finding(s) checked.")
PYEOF
- name: Post review comment
if: always() && steps.context.outputs.diff_size != '0'
shell: bash