Release Management Workflow

L3
ModelContextProtocolGithubHarmony

Implement comprehensive release management workflow including bug fixes, version updates, changelog creation, and PR merging.

Created by Zijian Wu
2025-08-15
Release CoordinationPr Workflows

Model Ranking

Click on the dots to view the trajectory of each task run
Model
Run Results
Pass@4
Pass^4
Avg Time
Avg Turns
Input Tokens
Output Tokens
Total Tokens
Gemini
gemini-3-pro-high
4
/4
351.7s
15.0
1,040,622
19,329
1,059,951
Gemini
gemini-3-pro-low
4
/4
317.1s
15.5
1,059,243
19,054
1,078,297
Grok
grok-4
2
/4
466.9s
29.0
1,673,975
2,965
1,688,587
Claude
claude-sonnet-4
1
/4
455.3s
23.8
1,518,726
10,147
1,528,873
Qwen
qwen-3-coder-plus
1
/4
217.7s
25.3
1,598,002
9,207
1,607,209
Claude
claude-opus-4-1
0
/1
--
565.0s
17.0
1,036,903
8,321
1,045,224
Claude
claude-opus-4-5-high
0
/4
1942.7s
73.5
6,003,459
130,273
6,133,732
Claude
claude-sonnet-4-5
0
/4
240.5s
19.3
1,375,621
10,229
1,385,849
Claude
claude-sonnet-4-high
0
/4
347.4s
25.0
1,622,981
14,019
1,637,000
Claude
claude-sonnet-4-low
0
/4
295.7s
23.0
1,453,895
10,662
1,464,557
DeepSeek
deepseek-chat
0
/4
436.8s
23.3
1,260,398
5,838
1,266,236
DeepSeek
deepseek-v3-1-terminus
0
/4
416.5s
13.0
784,241
4,666
788,907
DeepSeek
deepseek-v3-1-terminus-thinking
0
/4
1346.1s
16.3
893,219
28,597
921,816
DeepSeek
deepseek-v3-2-chat
0
/4
419.9s
28.3
1,488,059
7,865
1,495,923
DeepSeek
deepseek-v3-2-thinking
0
/4
604.1s
28.5
1,367,644
14,280
1,381,923
Gemini
gemini-2-5-flash
0
/4
255.5s
13.5
886,587
21,087
907,675
Gemini
gemini-2-5-pro
0
/4
176.2s
21.5
1,590,303
6,874
1,597,177
Z.ai
glm-4-5
0
/4
199.5s
12.5
652,655
4,139
656,793
OpenAI
gpt-4-1
0
/4
55.2s
8.3
291,470
1,340
292,810
OpenAI
gpt-4-1-mini
0
/4
126.5s
16.5
631,818
1,691
633,509
OpenAI
gpt-4-1-nano
0
/4
59.3s
12.8
638,783
1,419
640,202
OpenAI
gpt-5-high
0
/4
2689.2s
41.0
2,491,628
116,782
2,608,411
OpenAI
gpt-5-low
0
/4
543.4s
14.3
872,763
31,047
903,809
OpenAI
gpt-5-medium
0
/4
1072.4s
26.8
1,317,914
54,815
1,372,729
OpenAI
gpt-5-mini-high
0
/4
187.3s
22.0
891,378
10,798
902,177
OpenAI
gpt-5-mini-low
0
/4
67.8s
13.3
499,726
1,676
501,402
OpenAI
gpt-5-mini-medium
0
/4
108.7s
15.5
686,969
6,221
693,189
OpenAI
gpt-5-nano-high
0
/4
213.7s
15.3
880,069
32,073
912,142
OpenAI
gpt-5-nano-low
0
/4
62.0s
9.0
325,578
4,112
329,690
OpenAI
gpt-5-nano-medium
0
/4
236.8s
22.3
1,247,417
30,822
1,278,239
OpenAI
gpt-oss-120b
0
/4
21.0s
4.5
152,027
981
153,007
Grok
grok-4-fast
0
/4
102.4s
22.8
1,014,847
7,300
1,022,147
Grok
grok-code-fast-1
0
/4
467.1s
34.5
1,860,620
14,013
1,874,633
MoonshotAI
kimi-k2-0711
0
/4
571.4s
25.8
1,324,481
6,751
1,331,232
MoonshotAI
kimi-k2-0905
0
/4
1484.5s
60.3
3,718,587
11,961
3,730,548
OpenAI
o3
0
/4
150.8s
12.5
376,488
6,604
383,092
OpenAI
o4-mini
0
/4
169.0s
7.5
249,455
9,206
258,661
Qwen
qwen-3-max
0
/4
408.6s
52.0
4,168,802
4,296
4,173,099

Task State


Instruction

I need help implementing a comprehensive release management workflow for this harmony repository. Here's what I need you to do:

Step 1: Analyze Current State First, analyze the current open pull requests to understand what changes they contain and their impact on the codebase.

Step 2: Create Release Branch Create a release preparation branch called 'release-v1.1.0' from the current main branch.

Step 3: Apply Critical Bug Fixes On the release branch, apply the MetaSep token fix from PR #25 by creating/updating the file src/encoding.rs with the corrected content where FormattingToken::MetaSep maps to "<|meta_sep|>" instead of "<|channel|>".

Also create/update src/registry.rs to include the missing MetaSep and MetaEnd token registrations:

Rust
(FormattingToken::MetaSep, "<|meta_sep|>"),
(FormattingToken::MetaEnd, "<|meta_end|>"),

Step 4: Add Missing Utility File From PR #26, create the missing shadcn utils file demo/harmony-demo/src/lib/utils.ts with content:

TypeScript
import { clsx, type ClassValue } from "clsx"
import { twMerge } from "tailwind-merge"

export function cn(...inputs: ClassValue[]) {
  return twMerge(clsx(inputs))
}

And create/update .gitignore to add:

Plaintext
# Avoid ignoring shadcn utils
!demo/harmony-demo/src/lib

Step 5: Version Update Update the version number in Cargo.toml: Change the version field in the [package] section to version = "1.1.0".

Step 6: Create Comprehensive Changelog Create a CHANGELOG.md file in the release branch with the following content:

Markdown
# Changelog

## [1.1.0] - 2025-08-07

### Added
- Added missing shadcn utils.ts file for demo application
- Enhanced gitignore rules to preserve shadcn utilities

### Fixed
- Fixed MetaSep token mapping bug (was incorrectly mapped to channel token)
- Added missing MetaSep and MetaEnd token registrations in registry
- Improved tokenizer registry functionality for meta formatting tokens

### Changed
- Updated version to 1.1.0 for new release cycle

### Technical Details
- MetaSep token now correctly maps to `<|meta_sep|>` instead of `<|channel|>`
- Registry now properly recognizes MetaSep and MetaEnd formatting tokens
- Demo application now includes required utility functions for UI components

Step 7: Create Release Pull Request Create a pull request from 'release-v1.1.0' to 'main' with title "Release v1.1.0 - Bug fixes and utility additions" and a detailed description explaining all the integrated changes.

Step 8: Merge the Pull Request After creating the PR, merge it into the main branch using the "squash and merge" method.

Step 9: Verification Ensure the release branch contains at least 4 distinct commits before merging:

  1. MetaSep token fix commit
  2. Utility file addition commit
  3. Version update commit
  4. Changelog addition commit


Verify

*.py
Python
import sys
import os
import requests
from typing import Dict, List, Optional, Tuple
import base64
from dotenv import load_dotenv


def _get_github_api(
    endpoint: str, headers: Dict[str, str], org: str, repo: str = "harmony"
) -> Tuple[bool, Optional[Dict]]:
    """Make a GET request to GitHub API and return (success, response)."""
    url = f"https://api.github.com/repos/{org}/{repo}/{endpoint}"
    try:
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            return True, response.json()
        elif response.status_code == 404:
            return False, None
        else:
            print(f"API error for {endpoint}: {response.status_code}", file=sys.stderr)
            return False, None
    except Exception as e:
        print(f"Exception for {endpoint}: {e}", file=sys.stderr)
        return False, None


def _check_branch_exists(
    branch_name: str, headers: Dict[str, str], org: str, repo: str = "harmony"
) -> bool:
    """Verify that a branch exists in the repository."""
    success, _ = _get_github_api(f"branches/{branch_name}", headers, org, repo)
    return success


def _check_file_content(
    branch: str,
    file_path: str,
    keywords: List[str],
    headers: Dict[str, str],
    org: str,
    repo: str = "harmony",
) -> bool:
    """Verify that a file exists in branch and contains required keywords."""
    success, result = _get_github_api(
        f"contents/{file_path}?ref={branch}", headers, org, repo
    )
    if not success or not result:
        return False

    if keywords and result.get("content"):
        try:
            content = base64.b64decode(result.get("content", "")).decode("utf-8")
            return all(keyword in content for keyword in keywords)
        except Exception as e:
            print(f"Content decode error for {file_path}: {e}", file=sys.stderr)
            return False

    return True


def _check_specific_file_content(
    branch: str,
    file_path: str,
    expected_content: str,
    headers: Dict[str, str],
    org: str,
    repo: str = "harmony",
    min_length: int = 100,
) -> bool:
    """Verify that a file contains specific exact content and has reasonable size."""
    success, result = _get_github_api(
        f"contents/{file_path}?ref={branch}", headers, org, repo
    )
    if not success or not result:
        return False

    if result.get("content"):
        try:
            content = base64.b64decode(result.get("content", "")).decode("utf-8")
            # Check both that expected content exists and file has reasonable content
            return expected_content in content and len(content) >= min_length
        except Exception as e:
            print(f"Content decode error for {file_path}: {e}", file=sys.stderr)
            return False

    return False


def _check_pr_merged(
    title_substring: str,
    base_branch: str,
    headers: Dict[str, str],
    org: str,
    repo: str = "harmony",
) -> Tuple[bool, Optional[int]]:
    """Check if a PR with specified title was merged into base branch and return PR number."""
    # Check closed PRs to find merged ones
    success, prs = _get_github_api(
        "pulls?state=closed&per_page=100", headers, org, repo
    )
    if not success or not prs:
        return False, None

    for pr in prs:
        title_match = title_substring.lower() in pr.get("title", "").lower()
        base_match = pr.get("base", {}).get("ref") == base_branch
        is_merged = pr.get("merged_at") is not None

        if title_match and base_match and is_merged:
            return True, pr.get("number")

    return False, None


def _check_pr_squash_merged(
    pr_number: int, headers: Dict[str, str], org: str, repo: str = "harmony"
) -> bool:
    """Check if a PR was merged using squash and merge method."""
    # Get the PR details
    success, pr = _get_github_api(f"pulls/{pr_number}", headers, org, repo)
    if not success or not pr:
        return False

    if not pr.get("merged_at"):
        return False

    merge_commit_sha = pr.get("merge_commit_sha")
    if not merge_commit_sha:
        return False

    # Get the merge commit details
    success, commit = _get_github_api(f"commits/{merge_commit_sha}", headers, org, repo)
    if not success or not commit:
        return False

    # For squash and merge, the commit will have exactly one parent
    # and the commit message typically includes the PR number
    parents = commit.get("parents", [])
    commit_message = commit.get("commit", {}).get("message", "")

    # Squash and merge commits have exactly 1 parent (the base branch)
    # Regular merge commits have 2 parents (base and head branches)
    if len(parents) == 1 and f"#{pr_number}" in commit_message:
        return True

    return False


def verify() -> bool:
    """
    Programmatically verify that the release management workflow meets the
    requirements described in description.md.
    """
    # Load environment variables from .mcp_env
    load_dotenv(".mcp_env")

    # Get GitHub token and org
    github_token = os.environ.get("MCP_GITHUB_TOKEN")
    github_org = os.environ.get("GITHUB_EVAL_ORG")

    if not github_token:
        print("Error: MCP_GITHUB_TOKEN environment variable not set", file=sys.stderr)
        return False

    if not github_org:
        print("Error: GITHUB_EVAL_ORG environment variable not set", file=sys.stderr)
        return False

    # Configuration constants
    RELEASE_BRANCH = "release-v1.1.0"

    # Expected content checks with minimum file sizes to ensure files aren't just stubs
    METASEP_FIX = 'FormattingToken::MetaSep => "<|meta_sep|>"'
    REGISTRY_FIX = '(FormattingToken::MetaSep, "<|meta_sep|>")'
    METAEND_FIX = '(FormattingToken::MetaEnd, "<|meta_end|>")'
    UTILS_CONTENT = "export function cn(...inputs: ClassValue[])"
    GITIGNORE_ADDITION = "!demo/harmony-demo/src/lib"
    VERSION_110 = 'version = "1.1.0"'

    CHANGELOG_KEYWORDS = [
        "## [1.1.0] - 2025-08-07",
        "MetaSep token mapping bug",
        "shadcn utils.ts file",
        "Fixed MetaSep token",
        "Registry now properly recognizes",
    ]

    headers = {
        "Authorization": f"token {github_token}",
        "Accept": "application/vnd.github.v3+json",
    }

    # Run verification checks
    print("Verifying GitHub release management workflow completion...")

    # 1. Check release branch exists
    print("1. Verifying release branch exists...")
    if not _check_branch_exists(RELEASE_BRANCH, headers, github_org):
        print(f"Error: Branch '{RELEASE_BRANCH}' not found", file=sys.stderr)
        return False

    # 2. Check MetaSep fix in encoding.rs (with min content length to ensure file wasn't gutted)
    print("2. Verifying MetaSep token fix in encoding.rs...")
    if not _check_specific_file_content(
        "main", "src/encoding.rs", METASEP_FIX, headers, github_org, min_length=500
    ):
        print(
            "Error: MetaSep token fix not found in src/encoding.rs or file is too small",
            file=sys.stderr,
        )
        return False

    # 3. Check registry updates (both MetaSep and MetaEnd)
    print("3. Verifying MetaSep and MetaEnd registry additions...")
    if not _check_specific_file_content(
        "main", "src/registry.rs", REGISTRY_FIX, headers, github_org, min_length=500
    ):
        print(
            "Error: MetaSep registry fix not found in src/registry.rs or file is too small",
            file=sys.stderr,
        )
        return False
    if not _check_specific_file_content(
        "main", "src/registry.rs", METAEND_FIX, headers, github_org, min_length=500
    ):
        print(
            "Error: MetaEnd registry fix not found in src/registry.rs", file=sys.stderr
        )
        return False

    # 4. Check utils.ts file exists with correct content
    print("4. Verifying shadcn utils.ts file...")
    if not _check_specific_file_content(
        "main",
        "demo/harmony-demo/src/lib/utils.ts",
        UTILS_CONTENT,
        headers,
        github_org,
        min_length=50,
    ):
        print("Error: utils.ts file not found or incorrect content", file=sys.stderr)
        return False

    # 5. Check .gitignore update
    print("5. Verifying .gitignore update...")
    if not _check_specific_file_content(
        "main", ".gitignore", GITIGNORE_ADDITION, headers, github_org, min_length=100
    ):
        print("Error: .gitignore update not found", file=sys.stderr)
        return False

    # 6. Check version update in Cargo.toml only (pyproject.toml uses dynamic versioning)
    print("6. Verifying version update in Cargo.toml...")
    if not _check_specific_file_content(
        "main", "Cargo.toml", VERSION_110, headers, github_org, min_length=200
    ):
        print("Error: Version 1.1.0 not found in Cargo.toml", file=sys.stderr)
        return False

    # 7. Check CHANGELOG.md exists with required content
    print("7. Verifying CHANGELOG.md...")
    if not _check_file_content(
        "main", "CHANGELOG.md", CHANGELOG_KEYWORDS, headers, github_org
    ):
        print(
            "Error: CHANGELOG.md not found or missing required content", file=sys.stderr
        )
        return False

    # 8. Check release PR was merged and get PR number
    print("8. Verifying release pull request was merged...")
    pr_merged, pr_number = _check_pr_merged(
        "Release v1.1.0", "main", headers, github_org
    )
    if not pr_merged:
        print("Error: Release pull request not found or not merged", file=sys.stderr)
        return False

    # 9. Check PR was merged using squash and merge
    print("9. Verifying pull request was merged using 'squash and merge' method...")
    if pr_number and not _check_pr_squash_merged(pr_number, headers, github_org):
        print(
            f"Error: Pull request #{pr_number} was not merged using 'squash and merge' method",
            file=sys.stderr,
        )
        return False

    print("\n✓ All verification checks passed!")
    print("Release management workflow completed successfully.")
    return True


if __name__ == "__main__":
    success = verify()
    sys.exit(0 if success else 1)