Fix Conflict

L3
ModelContextProtocolGithubHarmony

Resolve merge conflicts by creating missing infrastructure and ensuring conflicted PRs can merge cleanly.

Created by Zijian Wu
2025-08-15
Pr Workflows

Model Ranking

Click on the dots to view the trajectory of each task run
Model
Run Results
Pass@4
Pass^4
Avg Time
Avg Turns
Input Tokens
Output Tokens
Total Tokens
Claude
claude-opus-4-5-high
4
/4
229.1s
19.8
1,762,839
7,381
1,770,220
OpenAI
gpt-5-high
3
/4
2114.3s
34.0
2,403,723
67,416
2,471,139
OpenAI
gpt-5-low
3
/4
296.8s
17.5
1,143,982
12,409
1,156,391
OpenAI
gpt-5-medium
3
/4
769.8s
27.5
1,805,184
30,597
1,835,781
DeepSeek
deepseek-v3-2-thinking
2
/4
424.5s
27.8
1,782,564
8,467
1,791,031
Z.ai
glm-4-5
2
/4
317.3s
27.0
1,792,334
5,838
1,798,172
OpenAI
gpt-5-mini-high
2
/4
409.0s
32.8
2,395,128
26,522
2,421,649
Claude
claude-sonnet-4-high
1
/4
243.2s
23.0
1,739,543
5,224
1,744,767
Gemini
gemini-3-pro-high
1
/4
418.6s
17.5
1,127,406
7,433
1,134,839
MoonshotAI
kimi-k2-0905
1
/4
640.4s
31.8
1,768,074
3,602
1,771,676
OpenAI
o4-mini
1
/4
552.0s
29.3
1,555,274
17,374
1,572,648
Qwen
qwen-3-coder-plus
1
/4
369.1s
29.5
2,141,636
3,592
2,145,228
Claude
claude-opus-4-1
0
/1
--
382.3s
14.0
869,307
4,839
874,146
Claude
claude-sonnet-4
0
/4
393.9s
24.8
1,975,114
8,267
1,983,382
Claude
claude-sonnet-4-5
0
/4
279.5s
28.0
3,087,341
6,525
3,093,866
Claude
claude-sonnet-4-low
0
/4
268.6s
25.8
2,021,594
5,297
2,026,891
DeepSeek
deepseek-chat
0
/4
371.9s
19.8
1,351,906
3,460
1,355,365
DeepSeek
deepseek-v3-1-terminus
0
/4
329.3s
12.0
820,762
1,530
822,292
DeepSeek
deepseek-v3-1-terminus-thinking
0
/4
363.1s
9.5
626,807
4,480
631,288
DeepSeek
deepseek-v3-2-chat
0
/4
332.0s
27.5
1,707,282
5,556
1,712,838
Gemini
gemini-2-5-flash
0
/4
53.2s
8.0
597,082
5,518
602,600
Gemini
gemini-2-5-pro
0
/4
96.3s
3.3
22,421
8,047
30,468
Gemini
gemini-3-pro-low
0
/4
236.9s
18.5
1,151,537
8,266
1,159,803
OpenAI
gpt-4-1
0
/4
80.3s
17.3
888,690
1,307
889,997
OpenAI
gpt-4-1-mini
0
/4
85.2s
15.8
904,255
858
905,112
OpenAI
gpt-4-1-nano
0
/4
45.9s
11.0
725,859
656
726,515
OpenAI
gpt-5-mini-low
0
/4
36.4s
5.5
215,443
865
216,308
OpenAI
gpt-5-mini-medium
0
/4
135.3s
15.5
775,291
4,502
779,793
OpenAI
gpt-5-nano-high
0
/4
332.6s
24.0
1,952,033
44,381
1,996,414
OpenAI
gpt-5-nano-low
0
/4
79.3s
11.8
612,873
2,972
615,845
OpenAI
gpt-5-nano-medium
0
/4
151.7s
17.5
1,204,803
16,448
1,221,251
OpenAI
gpt-oss-120b
0
/4
13.6s
3.5
114,136
350
114,486
Grok
grok-4
0
/4
612.0s
22.0
1,775,878
1,797
1,787,467
Grok
grok-4-fast
0
/4
133.0s
18.8
1,333,597
8,424
1,342,021
Grok
grok-code-fast-1
0
/4
126.6s
29.3
1,822,113
10,216
1,832,329
MoonshotAI
kimi-k2-0711
0
/4
306.1s
21.3
1,255,569
2,810
1,258,379
OpenAI
o3
0
/4
142.7s
16.8
821,691
4,236
825,927
Qwen
qwen-3-max
0
/4
191.5s
25.8
1,544,520
2,252
1,546,772

Task State


Instruction

I have some pull requests that won't merge due to conflicts. Can you help me fix the merge conflicts by creating the missing infrastructure?

Step 1: Find Conflicted PR Look through the open pull requests and find the one that has mergeable: false and mergeable_state: "dirty". Check what file it's trying to modify - it appears to be missing a file that the PR is trying to add or modify.

Step 2: Create Infrastructure PR
Create a new branch and PR to add the missing file that the conflicted PR needs. The PR must have:

  • Title: Must contain "Add CI infrastructure" and "resolve conflicts"
  • Body: Must include:
    • Reference to the conflicted PR using "Fixes #[PR_NUMBER]" or "Resolves #[PR_NUMBER]"
    • Explanation that this "prepares infrastructure" for the other PR
    • Mention of "missing .github directory" and "workflow conflicts"
  • File Content: Extract the complete file content from the conflicted PR's changes and add it to main. This ensures the conflicted PR can merge cleanly without conflicts.

Step 3: Merge Infrastructure PR Merge the infrastructure PR to main.

Step 4: Add Comment to Original PR Add a comment to the original conflicted PR that references the infrastructure PR you just created and merged. The comment must mention the infrastructure PR number using "PR #[NUMBER]" format.

Step 5: Merge Original PR Now merge the original conflicted PR since it should be able to merge cleanly.



Verify

*.py
Python
import sys
import os
import requests
from typing import Dict, Optional, Tuple
from dotenv import load_dotenv


def _get_github_api(
    endpoint: str, headers: Dict[str, str], org: str, repo: str = "harmony"
) -> Tuple[bool, Optional[Dict]]:
    """Make a GET request to GitHub API and return (success, response)."""
    url = f"https://api.github.com/repos/{org}/{repo}/{endpoint}"
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            return True, response.json()
        elif response.status_code == 404:
            return False, None
        else:
            print(f"API error for {endpoint}: {response.status_code}", file=sys.stderr)
            return False, None
    except Exception as e:
        print(f"Exception for {endpoint}: {e}", file=sys.stderr)
        return False, None


def _check_ci_file_exists(
    file_path: str, headers: Dict[str, str], org: str, repo: str = "harmony"
) -> bool:
    """Check if CI file exists in main branch."""
    success, _ = _get_github_api(f"contents/{file_path}?ref=main", headers, org, repo)
    return success


def _check_pr_comments(
    pr_number: int,
    infra_pr_number: int,
    headers: Dict[str, str],
    org: str,
    repo: str = "harmony",
) -> bool:
    """Check if PR has a comment linking to the infrastructure PR using 'PR #[NUMBER]' format."""
    success, comments = _get_github_api(
        f"issues/{pr_number}/comments", headers, org, repo
    )
    if not success or not comments:
        return False

    # Look for "PR #123" pattern (case insensitive)
    import re

    for comment in comments:
        body = comment.get("body", "")
        if re.search(rf"PR\s*#{infra_pr_number}", body, re.IGNORECASE):
            return True
    return False


def _find_infrastructure_pr(
    headers: Dict[str, str], org: str, repo: str = "harmony"
) -> Optional[Dict]:
    """Find the infrastructure PR by checking title and body content."""
    success, prs = _get_github_api("pulls?state=all&per_page=50", headers, org, repo)
    if success and prs:
        for pr in prs:
            title = pr.get("title", "").lower()
            body = pr.get("body", "").lower()

            # Check title contains required keywords
            title_ok = "add ci infrastructure" in title and "resolve conflicts" in title

            # Check body contains required elements
            has_reference = "fixes #" in body or "resolves #" in body
            has_prep_text = "prepares infrastructure" in body
            has_github_text = "missing .github directory" in body
            has_workflow_text = "workflow conflicts" in body

            body_ok = (
                has_reference
                and has_prep_text
                and has_github_text
                and has_workflow_text
            )

            if title_ok and body_ok:
                return pr
    return None


def verify() -> bool:
    """
    Programmatically verify that the merge conflict resolution workflow meets the
    requirements described in description.md.
    """
    # Load environment variables from .mcp_env
    load_dotenv(".mcp_env")

    # Get GitHub token and org
    github_token = os.environ.get("MCP_GITHUB_TOKEN")
    github_org = os.environ.get("GITHUB_EVAL_ORG")

    if not github_token:
        print("Error: MCP_GITHUB_TOKEN environment variable not set", file=sys.stderr)
        return False

    if not github_org:
        print("Error: GITHUB_EVAL_ORG environment variable not set", file=sys.stderr)
        return False

    headers = {
        "Authorization": f"token {github_token}",
        "Accept": "application/vnd.github.v3+json",
    }

    # Run verification checks
    print("Verifying merge conflict resolution workflow completion...")

    # 1. Check that CI infrastructure file exists in main (extracted from conflicted PR)
    print("1. Checking CI infrastructure was added to main...")
    # Check for both CI.yml and ci.yml (case-insensitive)
    ci_exists = _check_ci_file_exists(".github/workflows/CI.yml", headers, github_org)
    if not ci_exists:
        ci_exists = _check_ci_file_exists(".github/workflows/ci.yml", headers, github_org)
    
    if not ci_exists:
        print("Error: Neither .github/workflows/CI.yml nor .github/workflows/ci.yml found in main", file=sys.stderr)
        return False

    # 2. Find infrastructure PR with required title and body content
    print("2. Finding infrastructure PR with required content...")
    infra_pr = _find_infrastructure_pr(headers, github_org)
    if not infra_pr:
        print(
            "Error: No infrastructure PR found with required title and body content",
            file=sys.stderr,
        )
        print(
            "Required title: 'Add CI infrastructure' and 'resolve conflicts'",
            file=sys.stderr,
        )
        print(
            "Required body: reference with 'Fixes #' or 'Resolves #', 'prepares infrastructure', 'missing .github directory', 'workflow conflicts'",
            file=sys.stderr,
        )
        return False

    print(f"Found infrastructure PR #{infra_pr.get('number')}: {infra_pr.get('title')}")

    # 3. Check that infrastructure PR is merged
    if not infra_pr.get("merged_at"):
        print(
            f"Error: Infrastructure PR #{infra_pr.get('number')} not merged yet",
            file=sys.stderr,
        )
        return False

    # 4. Check that PR #24 is merged
    print("3. Checking that PR #24 is merged...")
    success, pr24 = _get_github_api("pulls/24", headers, github_org)
    if not success or not pr24:
        print("Error: PR #24 not found", file=sys.stderr)
        return False

    if not pr24.get("merged_at"):
        print("Error: PR #24 is not merged yet", file=sys.stderr)
        return False

    # 5. Check that PR #24 has a comment linking to the infrastructure PR
    print("4. Checking that PR #24 has comment linking to infrastructure PR...")
    if not _check_pr_comments(24, infra_pr.get("number"), headers, github_org):
        print(
            f"Error: PR #24 missing comment linking to infrastructure PR #{infra_pr.get('number')}",
            file=sys.stderr,
        )
        return False

    print("\n✓ Task completed successfully!")
    print(
        f"Infrastructure PR #{infra_pr.get('number')} extracted content from PR #24 and resolved conflicts"
    )
    print(
        "PR #24 is now merged cleanly and has a comment linking to the infrastructure PR"
    )
    return True


if __name__ == "__main__":
    success = verify()
    sys.exit(0 if success else 1)