Advanced Branch Strategy

L3
ModelContextProtocolGithubEasyR1

Implement GitFlow branching strategy with develop, release, and feature branches to replace risky direct-to-main development.

Created by Xiangyan Liu
2025-08-15
Pr WorkflowsRelease Coordination

Model Ranking

Click on the dots to view the trajectory of each task run
Model
Run Results
Pass@4
Pass^4
Avg Time
Avg Turns
Input Tokens
Output Tokens
Total Tokens
Claude
claude-sonnet-4
4
/4
106.4s
10.3
221,317
2,411
223,727
Claude
claude-sonnet-4-5
4
/4
78.5s
9.5
211,117
2,291
213,408
Claude
claude-sonnet-4-high
4
/4
73.9s
10.0
194,569
2,585
197,154
Claude
claude-sonnet-4-low
4
/4
77.6s
10.0
192,863
2,368
195,231
DeepSeek
deepseek-chat
4
/4
161.9s
13.0
271,607
1,426
273,033
Gemini
gemini-2-5-flash
4
/4
36.6s
10.0
210,150
2,370
212,520
Z.ai
glm-4-5
4
/4
98.0s
14.3
340,969
2,512
343,481
OpenAI
gpt-5-high
4
/4
281.5s
11.0
125,157
12,022
137,180
OpenAI
gpt-5-low
4
/4
141.2s
12.8
228,818
6,357
235,175
OpenAI
gpt-5-medium
4
/4
150.2s
10.8
122,328
5,951
128,279
Grok
grok-4
4
/4
197.2s
6.8
150,138
945
157,068
Grok
grok-code-fast-1
4
/4
45.4s
10.3
170,398
2,250
172,648
MoonshotAI
kimi-k2-0905
4
/4
157.4s
10.0
145,380
1,562
146,941
Qwen
qwen-3-coder-plus
4
/4
60.0s
12.8
496,106
1,552
497,658
Claude
claude-opus-4-5-high
3
/4
109.9s
12.3
183,402
3,490
186,892
DeepSeek
deepseek-v3-1-terminus-thinking
3
/4
451.1s
8.0
186,874
10,110
196,984
Gemini
gemini-2-5-pro
3
/4
71.4s
6.3
132,992
4,093
137,085
OpenAI
gpt-4-1
3
/4
39.2s
9.3
107,370
882
108,252
OpenAI
gpt-4-1-mini
3
/4
49.4s
11.0
137,317
874
138,191
OpenAI
gpt-5-mini-low
3
/4
42.4s
10.0
146,074
942
147,016
OpenAI
gpt-5-nano-high
3
/4
92.5s
11.3
155,122
11,779
166,901
OpenAI
gpt-5-nano-medium
3
/4
78.0s
12.8
152,031
8,214
160,245
OpenAI
gpt-oss-120b
3
/4
38.4s
10.0
112,533
1,847
114,380
Grok
grok-4-fast
3
/4
60.2s
7.5
130,521
4,236
134,757
MoonshotAI
kimi-k2-0711
3
/4
136.8s
9.8
176,982
1,370
178,352
Qwen
qwen-3-max
3
/4
57.5s
11.3
208,297
828
209,125
DeepSeek
deepseek-v3-2-thinking
2
/4
270.6s
19.3
346,354
6,009
352,363
Gemini
gemini-3-pro-low
2
/4
184.1s
16.8
1,016,029
6,501
1,022,530
OpenAI
gpt-5-mini-high
2
/4
99.2s
15.0
245,057
4,644
249,701
OpenAI
gpt-5-mini-medium
2
/4
57.9s
12.8
169,476
2,392
171,867
OpenAI
o3
2
/4
63.6s
10.3
150,754
2,332
153,086
OpenAI
o4-mini
2
/4
182.8s
10.5
150,809
9,540
160,349
Claude
claude-opus-4-1
1
/1
--
198.9s
10.0
235,867
2,372
238,239
DeepSeek
deepseek-v3-1-terminus
1
/4
61.8s
3.0
61,687
855
62,541
DeepSeek
deepseek-v3-2-chat
1
/4
235.6s
22.8
722,204
4,079
726,283
Gemini
gemini-3-pro-high
1
/4
281.7s
20.0
933,249
7,193
940,442
OpenAI
gpt-4-1-nano
0
/4
44.0s
12.3
143,551
845
144,397
OpenAI
gpt-5-nano-low
0
/4
54.5s
11.5
134,790
2,538
137,328

Task State


Instruction

The EasyR1 repository has a critical production issue: all development happens directly on the main branch, which is extremely risky for a project with 25 active issues. A recent commit 098931530606d22f867fd121b1dcb3225a43661f introduced protocol changes that need to be properly managed through a structured branching workflow. I need you to implement a complete GitFlow strategy by working through a realistic development scenario.

The Scenario: You're preparing for the v1.0.0 release while simultaneously handling a critical protocol serialization bug that was introduced in the recent data proto changes.

Step 1: Initialize GitFlow Structure Create a develop branch from main as the new integration branch. Then create a release/v1.0.0 branch from develop to prepare for the upcoming release.

Step 2: Address the Critical Bug Create a feature/protocol-serialization-fix branch from develop. In this branch, create a new file called PROTOCOL_FIXES.md with the exact content:

Plaintext
# Protocol Serialization Fixes

## Critical Fix for Data Proto Issue
- Enhanced serialization safety check implemented
- Addresses issue from commit 098931530606d22f867fd121b1dcb3225a43661f
- Status: Ready for integration testing

Step 3: Integrate the Fix Through Proper Workflow Create a pull request from feature/protocol-serialization-fix to develop to integrate the fix documentation. This demonstrates the feature → develop integration pattern.

Step 4: Update Release Branch and CI/CD Merge the develop branch changes into release/v1.0.0 branch to include the critical fix in the release.

Step 5: Document the New Process Create an issue titled Implement Advanced Branch Protection Strategy with exactly these 3 checkboxes in the body:

  • All development flows through develop branch
  • Release preparation happens in release/v1.0.0 branch
  • Feature integration uses PR workflow

Add the label process-implementation to this issue to track the process implementation.



Verify

*.py
Python
import sys
import os
import requests
from typing import Dict, Optional, Tuple
from dotenv import load_dotenv

load_dotenv(".mcp_env")


def _get_github_api(
    endpoint: str, headers: Dict[str, str]
) -> Tuple[bool, Optional[Dict]]:
    """Make a GET request to GitHub API and return (success, response)."""
    github_org = os.environ.get("GITHUB_EVAL_ORG")
    url = f"https://api.github.com/repos/{github_org}/EasyR1/{endpoint}"
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            return True, response.json()
        elif response.status_code == 404:
            return False, None
        else:
            print(f"API error for {endpoint}: {response.status_code}", file=sys.stderr)
            return False, None
    except Exception as e:
        print(f"Exception for {endpoint}: {e}", file=sys.stderr)
        return False, None


def _check_gitflow_branches(headers: Dict[str, str]) -> bool:
    """Check if GitFlow branches are properly created from correct base branches."""
    success, branches_data = _get_github_api("branches", headers)
    if not success or not branches_data:
        print("Error: Could not fetch branches", file=sys.stderr)
        return False

    existing_branches = [branch.get("name", "") for branch in branches_data]
    required_branches = [
        "develop",
        "release/v1.0.0",
        "feature/protocol-serialization-fix",
    ]

    for branch in required_branches:
        if branch not in existing_branches:
            print(f"Error: Required branch '{branch}' not found", file=sys.stderr)
            return False

    return True


def _check_protocol_fixes_file(headers: Dict[str, str]) -> bool:
    """Check if PROTOCOL_FIXES.md file exists in feature branch with correct content."""
    success, file_data = _get_github_api(
        "contents/PROTOCOL_FIXES.md?ref=feature/protocol-serialization-fix", headers
    )
    if not success or not file_data:
        print("Error: PROTOCOL_FIXES.md not found in feature branch", file=sys.stderr)
        return False

    # Decode base64 content
    import base64

    content = base64.b64decode(file_data.get("content", "")).decode("utf-8")

    # Check for required content elements
    required_elements = [
        "# Protocol Serialization Fixes",
        "## Critical Fix for Data Proto Issue",
        "Enhanced serialization safety check implemented",
        "098931530606d22f867fd121b1dcb3225a43661f",
        "Status: Ready for integration testing",
    ]

    for element in required_elements:
        if element not in content:
            print(
                f"Error: PROTOCOL_FIXES.md missing required content: {element}",
                file=sys.stderr,
            )
            return False

    return True


def _check_integration_workflow(headers: Dict[str, str]) -> Optional[Dict]:
    """Verify the feature → develop integration pull request exists."""
    # Check both open and closed PRs since the workflow may have completed
    success, prs = _get_github_api("pulls?state=all", headers)
    if not success or not prs:
        print("Error: Could not fetch pull requests", file=sys.stderr)
        return None

    for pr in prs:
        head_ref = pr.get("head", {}).get("ref", "")
        base_ref = pr.get("base", {}).get("ref", "")

        if head_ref == "feature/protocol-serialization-fix" and base_ref == "develop":
            return pr

    print(
        "Error: Integration PR from feature/protocol-serialization-fix to develop not found",
        file=sys.stderr,
    )
    return None


def _check_release_branch_updated(headers: Dict[str, str]) -> bool:
    """Check if release branch contains the develop branch changes."""
    # Check if PROTOCOL_FIXES.md exists in release branch
    success, file_data = _get_github_api(
        "contents/PROTOCOL_FIXES.md?ref=release/v1.0.0", headers
    )
    if not success or not file_data:
        print(
            "Error: PROTOCOL_FIXES.md not found in release branch - develop changes not merged",
            file=sys.stderr,
        )
        return False

    return True


def _check_process_documentation(headers: Dict[str, str]) -> Optional[Dict]:
    """Check if process is properly documented in an issue."""
    success, issues = _get_github_api("issues", headers)
    if not success or not issues:
        print("Error: Could not fetch issues for documentation check", file=sys.stderr)
        return None

    expected_title = "Implement Advanced Branch Protection Strategy"
    expected_checkboxes = [
        "All development flows through develop branch",
        "Release preparation happens in release/v1.0.0 branch",
        "Feature integration uses PR workflow",
    ]

    for issue in issues:
        title = issue.get("title", "")
        if title == expected_title:
            body = issue.get("body", "")

            # Check for exactly 3 checkboxes with specific content
            checkbox_count = body.count("- [ ]") + body.count("- [x]")
            if checkbox_count != 3:
                print(
                    f"Error: Documentation issue should have 3 checkboxes, found {checkbox_count}",
                    file=sys.stderr,
                )
                return None

            # Check for specific checkbox content
            for expected_text in expected_checkboxes:
                if expected_text not in body:
                    print(
                        f"Error: Documentation issue missing required checkbox: {expected_text}",
                        file=sys.stderr,
                    )
                    return None

            # Check label assignment
            labels = issue.get("labels", [])
            label_names = [label.get("name") for label in labels]
            if "process-implementation" not in label_names:
                print(
                    "Error: Documentation issue not labeled with 'process-implementation'",
                    file=sys.stderr,
                )
                return None

            return issue

    print("Error: Process documentation issue not found", file=sys.stderr)
    return None


def verify() -> bool:
    """
    Verify the complete GitFlow implementation following the integrated workflow
    described in description.md.
    """
    # Get GitHub token
    github_token = os.environ.get("MCP_GITHUB_TOKEN")
    if not github_token:
        print("Error: MCP_GITHUB_TOKEN environment variable not set", file=sys.stderr)
        return False

    headers = {
        "Authorization": f"token {github_token}",
        "Accept": "application/vnd.github.v3+json",
    }

    print("Verifying integrated GitFlow workflow implementation...")

    # 1. Verify GitFlow structure initialization
    print("1. Checking GitFlow branch structure...")
    if not _check_gitflow_branches(headers):
        return False

    # 2. Verify critical bug fix implementation via new file
    print("2. Checking protocol serialization fix documentation...")
    if not _check_protocol_fixes_file(headers):
        return False

    # 3. Verify integration workflow (feature → develop PR)
    print("3. Checking feature integration workflow...")
    integration_pr = _check_integration_workflow(headers)
    if not integration_pr:
        return False

    # 4. Verify release branch updated and CI configured
    print("4. Checking release branch sync and CI configuration...")
    if not _check_release_branch_updated(headers):
        return False

    # 5. Verify process documentation
    print("5. Checking process documentation...")
    doc_issue = _check_process_documentation(headers)
    if not doc_issue:
        return False

    print("\n✓ Integrated GitFlow workflow successfully implemented!")
    print("✓ GitFlow structure: main → develop → release/v1.0.0 branches created")
    print("✓ Critical fix: Protocol fix documented in PROTOCOL_FIXES.md file")
    print(
        f"✓ Integration: PR #{integration_pr.get('number')} demonstrates feature → develop workflow"
    )
    print(
        "✓ Release prep: Release branch contains develop changes, CI configured for both branches"
    )
    print(
        f"✓ Documentation: Process documented in issue #{doc_issue.get('number')} with proper checkboxes"
    )
    print(
        "\nThe repository now has a structured GitFlow workflow ready for implementation!"
    )
    return True


if __name__ == "__main__":
    success = verify()
    sys.exit(0 if success else 1)