release-config-self-check: Add a release configuration self-check #3
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "task/release-config-self-check"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Assigned idea
Benchmark authoring specification.
Summary: Add a dependency-free Node validation script that protects the existing release-please Node release policy for rendobar-cli.
Task intent: Maintain the repo's release automation by adding a small upstream-facing check that validates release-please-config.json continues to use the intended single-package Node release settings.
Benchmark plan: Construct the task package from rendobar/cli at f37e25a72284ecb9a1e04601d1126028be76759e. The solver-facing work request should ask for a new dependency-free Node script at scripts/verify-release-please-config.mjs that reads release-please-config.json and verifies the current policy: top-level release-type is node, include-component-in-tag is false, include-v-in-tag is true, packages['.'] exists with package-name rendobar-cli, changelog-path CHANGELOG.md, draft false, prerelease false, and release-type node, and the changelog section visibility matches the existing config. The task package should include hidden test_outputs.py checks and should not require submitting any upstream issue.
Environment plan: Environment/repo: use the checked-out repository at the fixed commit with Node.js and Python 3 available; no network access or dependency installation should be needed. Task/environment: provide only the ordinary maintainer-facing instructions plus the hidden Python verifier; the verifier runs from the repository root against the solver's working tree.
Verifier plan: test_outputs.py should assert that scripts/verify-release-please-config.mjs exists, run node scripts/verify-release-please-config.mjs and expect exit code 0 on the unmodified release-please-config.json, then copy the solved tree to temporary directories and mutate release-please-config.json for negative cases. Negative cases should include changing the top-level release-type away from node, changing packages['.'].release-type away from node, changing packages['.'].package-name away from rendobar-cli, and making a hidden changelog section visible; each mutation must make the command exit nonzero with a diagnostic mentioning release-please-config.json and the failing field.
Oracle plan: The oracle solution is a small ES module script that uses fs.readFileSync and JSON.parse, checks the documented invariants with explicit assertions, reports all failures or the first clear failure, and exits with process.exitCode = 1 on invalid config. It should keep release-please-config.json semantically unchanged.
Difficulty plan: Current coding agents may overfit to a success-only command, forget to test negative mutations, add package-manager dependencies, or validate only the top-level release-type while missing the package-level Node release policy and hidden changelog sections.
Repo patch scope: Expected upstream edits are confined to scripts/verify-release-please-config.mjs. Do not change CLI source directories or the semantic contents of release-please-config.json.
Anti-leakage notes: Do not expose solver-facing references to benchmark, harness, Snorkel, Harbor, std-ts, agent, stage, attempt, retry, generated task, or evaluation context.
Approach: Use release-please-config.json as the single source of truth and implement a deterministic local validator around its current Node release configuration.
Verification: Run python3 test_outputs.py; it checks scripts/verify-release-please-config.mjs and executes node scripts/verify-release-please-config.mjs against release-please-config.json plus temporary mutated configs.
Risk: Low implementation risk because the evidence file is plain JSON and the check can be dependency-free. Main risk is an overly broad or hardcoded script that does not actually parse the file; the negative verifier cases mitigate that.
Path rules: task package is tasks/; mutable upstream checkout is tasks//task/environment/repo; benchmark package files are under tasks//task; verifier/container environment belongs under tasks//task/environment; regular and UI verifier logic belongs in task/tests/test_outputs.py and test.sh only runs that file through pytest.
Agent context
Task slug: release-config-self-check
Branch: task/release-config-self-check
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.