labeler-table-rank-column: Add ranks to the Bluesky labeler table #1
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "task/labeler-table-rank-column"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Assigned idea
Benchmark authoring specification.
Summary: Create an upstream-facing README table update that adds a deterministic Rank column to the existing Bluesky labeler likes table while preserving the existing labeler rows and like counts.
Task intent: Modify the repository output so the README.md labeler table includes a Rank column before Labeler, with ranks assigned in the current descending Likes order shown in the file.
Benchmark plan: Build a regular task package from commit 76af112b87c87f37f4a1da626a24507ce951cfcd. The solver-facing prompt should ask for an ordinary repository change: update the Bluesky labelers table in README.md so the table header becomes Rank, Labeler, Likes and each existing row receives a 1-based rank matching its current order. Keep the Last updated line, table markers, profile links, HTML escaping, names, descriptions, and Likes values unchanged.
Environment plan: The task environment should contain the repository checkout and the normal file editing tools only; no network access is required because the expected work is based on the committed README.md contents. The benchmark environment should add a hidden verifier file such as test_outputs.py outside the solver-visible repo, plus any runner metadata needed to execute it.
Verifier plan: test_outputs.py should read README.md, locate the table between and if present or from the visible table header onward, and assert that the header is exactly | Rank | Labeler | Likes | followed by | --- | --- | --- |. It should parse every data row, require exactly three cells, require the first cell to be consecutive integers starting at 1, require the Likes column values to match the original committed sequence including 31103 for TTRPG Class Identifier, 8162 for Skywatch Blue, and 7141 for Perisai, and require the Labeler cell text to remain unchanged for all rows.
Oracle plan: The oracle solution is a minimal README.md edit: replace the two-column table header and separator with a three-column version and prefix each existing labeler row with its 1-based rank. The oracle should not refresh live Bluesky data or change the timestamp, links, descriptions, escaping, or like counts.
Difficulty plan: Current coding agents may overfit by re-scraping live data, rewriting the README with normalized Markdown, altering HTML entities such as & and |, changing timestamps, or sorting rows again instead of preserving the committed order. The hidden verifier should catch these plausible but incorrect edits.
Repo patch scope: Expected upstream edit scope is README.md only, specifically the labeler table header, separator, and row prefixes.
Anti-leakage notes: The solver-facing prompt and repository files must not mention benchmark, harness, Snorkel, Harbor, std-ts, agent, stage, attempt, retry, generated task, or evaluation context.
Approach: Use the committed README.md as the source of truth. Add a Rank column to the table and number the existing rows in place without changing any labeler content.
Verification: Run python test_outputs.py from the benchmark verifier directory after applying the solution; it must inspect README.md and pass all table structure, rank sequence, and unchanged-content assertions.
Risk: The main risk is that README.md may not include an explicit table-end marker; the verifier should handle that by parsing from through the contiguous Markdown table rows.
Path rules: task package is tasks/; mutable upstream checkout is tasks//task/environment/repo; benchmark package files are under tasks//task; verifier/container environment belongs under tasks//task/environment; regular and UI verifier logic belongs in task/tests/test_outputs.py and test.sh only runs that file through pytest.
Agent context
Task slug: labeler-table-rank-column
Branch: task/labeler-table-rank-column
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.