Skip to content

feat: group test cases by tag#112

Merged
kilinchange merged 5 commits intomasterfrom
feat/testcase_group
Mar 11, 2026
Merged

feat: group test cases by tag#112
kilinchange merged 5 commits intomasterfrom
feat/testcase_group

Conversation

@Chamberlain0w0
Copy link
Contributor

@Chamberlain0w0 Chamberlain0w0 commented Mar 11, 2026

  1. test_config.json:测试配置从单一平铺列表调整为按 tag 分组的 test_groups 结构,目前所有脚本的逻辑都是按组维护测例;
  2. run_models_and_profile.bash:测试日志和 profile 日志改为按 tag 分类存储到 logs/<tag>/ 以及 profile_logs/<tag> 中;新增了 --only-run 参数,可以按单个或多个 tag(半角逗号分隔)选择性执行测试;
  3. token.json:由于每个功能模块单开表格记录,所以每个 tag 对应一套表格,现在长这样:
{
    "APP_ID": "cli_XXXXXX",
    "APP_SECRET": "XXXXXX",
    "TAG_SPREADSHEET_CONFIGS": [
        {
            "tag": "basic",
            "MODEL_SPREADSHEET_TOKEN": {
                "GPT2": "XXXXXXXX",
                "LLAMA3": "XXXXXXXXX"
            }
        },
        {
            "tag": "zero",
            "MODEL_SPREADSHEET_TOKEN": {
                "GPT2": "XXXXXXXX",
                "LLAMA3": "XXXXXXXX"
            }
        }
    ]
}
  1. write_to_feishu_sheet.py:按 tag 读取 log,并写入对应的表格;同时由于服务器网络环境成谜,新增 retry 逻辑,如果同一请求连续 3 次超时,脚本会直接报 fatal 并退出,这时需要人工手动回退飞书表格历史版本。
  2. compare_xxx.py:适配新的日志目录结构,能够兼容带 tag 子目录的日志组织方式,逻辑改为对比最终的不带目录的纯文件名(test id),可以拿新的日志目录跟以前的旧的比。

@Chamberlain0w0
Copy link
Contributor Author

新的 log 目录现在是这样:
image

@Chamberlain0w0
Copy link
Contributor Author

image compare 脚本运行时,新的 log 目录与旧的 log 目录兼容,可以按老方法比较

@Chamberlain0w0
Copy link
Contributor Author

采用 --only-run 参数仅运行一个测例组
image

@Chamberlain0w0
Copy link
Contributor Author

后面两个 commit 之后的行为跟上述截图依然一致,就不再贴图了

import sys


def collect_log_files(base_dir: Path):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个是在 log 目录下面,递归收集可比较日志,不看 tag,只按 basename 索引

return files, duplicates


def exit_if_duplicate_logs(base_dir: Path, duplicates):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

由于只按 basename 索引,所以要求了不同 tag 下面的测例不能重名(我们目前也做到了这一点)

cat <<'EOF'
Usage: run_models_and_profile.bash [--test-config path] [--only-run tag1,tag2]

Options:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

加了两个参数

@kilinchange kilinchange merged commit b1e4b03 into master Mar 11, 2026
2 checks passed
@kilinchange kilinchange deleted the feat/testcase_group branch March 11, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants