Verify Tests Fail Without Fix

Verifies UI tests actually catch the issue. Supports two workflow modes:

Mode 1: Verify Failure Only (Test Creation)

Use when creating tests before writing a fix:

• Runs tests to verify they FAIL (proving they catch the bug)
• No fix files required
• Perfect for test-first development

# Auto-detect test filter from changed test files
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android

# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356"

Mode 2: Full Verification (Fix Validation)

Use when validating both tests and fix:

1. Without fix - tests should FAIL (bug is present)
2. With fix - tests should PASS (bug is fixed)

# Auto-detect everything (recommended)
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification

# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification

Note: -RequireFullVerification ensures the script errors if no fix files are detected, preventing silent fallback to failure-only mode.

Requirements

Verify Failure Only Mode:

• Test files in the PR (or working directory)

Full Verification Mode:

• Test files in the PR
• Fix files in the PR (non-test code changes)

The script auto-detects which mode to use based on whether fix files are present.

Expected Output

Verify Failure Only Mode:

╔═══════════════════════════════════════════════════════════╗
║              VERIFICATION PASSED ✅                       ║
╠═══════════════════════════════════════════════════════════╣
║  Tests FAILED as expected!                                ║
║  This proves the tests correctly reproduce the bug.       ║
╚═══════════════════════════════════════════════════════════╝

Full Verification Mode:

╔═══════════════════════════════════════════════════════════╗
║              VERIFICATION PASSED ✅                       ║
╠═══════════════════════════════════════════════════════════╣
║  - FAIL without fix (as expected)                         ║
║  - PASS with fix (as expected)                            ║
╚═══════════════════════════════════════════════════════════╝

What It Does

Verify Failure Only Mode (no fix files):

1. Fetches base branch from origin (if available)
2. Auto-detects test classes from changed test files
3. Runs tests (should FAIL to prove they catch the bug)
4. Updates PR labels based on result
5. Reports result

Full Verification Mode (fix files detected):

1. Fetches base branch from origin to ensure accurate diff
2. Auto-detects fix files (non-test code) from git diff
3. Auto-detects test classes from TestCases.Shared.Tests/*.cs
4. Reverts fix files to base branch
5. Runs tests (should FAIL without fix)
6. Restores fix files
7. Runs tests (should PASS with fix)
8. Generates markdown reports:
   • CustomAgentLogsTmp/TestValidation/verification-report.md - Full detailed report
   • CustomAgentLogsTmp/PRState/verification-report.md - Gate section for PR agent
9. Updates PR labels based on result
10. Reports result

PR Labels

The skill automatically manages two labels on the PR to indicate verification status:

Behavior:

• When verification passes, adds s/ai-reproduction-confirmed and removes s/ai-reproduction-failed if present
• When verification fails, adds s/ai-reproduction-failed and removes s/ai-reproduction-confirmed if present
• If a PR is re-verified after fixing tests, labels are updated accordingly
• No label = AI hasn't verified tests yet

Output Files

The skill generates output files under CustomAgentLogsTmp/PRState/<PRNumber>/verify-tests-fail/:

Plus UI test logs in CustomAgentLogsTmp/UITests/:

• android-device.log or ios-device.log - Device logs
• test-output.log - NUnit test output

Example structure:

CustomAgentLogsTmp/
├── UITests/                           # Shared UI test logs
│   ├── android-device.log
│   └── test-output.log
└── PRState/
    └── 27847/
        └── verify-tests-fail/
            ├── verification-report.md  # Full detailed report
            ├── verification-log.txt
            ├── test-without-fix.log
            └── test-with-fix.log

PR Number Detection:

• Auto-detected from branch name (e.g., pr-27847)
• Falls back to gh pr view command
• Uses "unknown" if detection fails
• Can be manually specified with -PRNumber parameter

Troubleshooting

Optional Parameters

# Require full verification (fail if no fix files detected) - recommended
-RequireFullVerification

# Explicit test filter
-TestFilter "Issue32030|ButtonUITests"

# Explicit fix files  
-FixFiles @("src/Core/src/File.cs")

# Explicit base branch
-BaseBranch "main"