notebook-message-summary-guardrails: Notebook Message Summary Guardrails #5
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "task/notebook-message-summary-guardrails"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Assigned idea
Benchmark authoring specification.
Summary: Improve the module 3.2 message-management notebook so its SummarizationMiddleware example has a deterministic, locally checkable preview of which Lunapolis messages are summarized versus retained before any model-backed call runs.
Task intent: Ask for ordinary upstream-facing maintenance of the educational notebook: make the existing message summarization lesson clearer and safer by extracting the sample HumanMessage/AIMessage history into a named variable and adding a small pure-Python preview helper for the configured keep=("messages", 1) behavior.
Benchmark plan: Construct the task package from the repository at the fixed commit. The solver prompt should target only notebooks/module-3/3.2_managing_messages.ipynb and request a deterministic preview cell placed near the existing "Summarize messages" section. The package should not require external LLM access. The expected work is to preserve the existing SummarizationMiddleware configuration trigger=("tokens", 100), keep=("messages", 1), reuse the existing Lunapolis conversation via a variable, and add a helper such as preview_summarization_window(messages, keep_messages=1) that returns the messages that would be summarized and the trailing messages that would be retained.
Environment plan: Task/environment: provide Python with nbformat and the repo's normal notebook dependencies sufficient to import langchain.messages, but do not provide API keys or network access. Environment/repo: materialize langchain-ai/lca-lc-foundations at commit 5e18afda75c51868d2e9857b86f20480cebadc04 and leave the notebook structure intact; hidden verifier files live outside the upstream tree.
Verifier plan: test_outputs.py should parse notebooks/module-3/3.2_managing_messages.ipynb with nbformat, locate the deterministic preview code, and execute only safe cells or extracted definitions without invoking create_agent or any model. It should assert that the notebook defines a named Lunapolis message list, the invoke call uses that variable, the middleware still contains trigger=("tokens", 100) and keep=("messages", 1), and preview_summarization_window on the notebook's message list returns seven summarized messages and one retained trailing message for the eight-message example, with message order and HumanMessage/AIMessage types preserved.
Oracle plan: The oracle solution is a patched copy of notebooks/module-3/3.2_managing_messages.ipynb that introduces a short markdown explanation plus a code cell defining the message list and preview_summarization_window, then updates the existing response = agent.invoke call to pass {"messages": lunapolis_messages}. The oracle should not add new dependencies or contact model services during verification.
Difficulty plan: Current coding systems may overfit to running the notebook and fail because API-backed cells cannot execute in the test environment. They may also edit notebook JSON incorrectly, change the educational middleware settings, inline the conversation instead of making it reusable, or implement a token counter rather than the requested deterministic keep-last-message preview.
Repo patch scope: Expected upstream edits are confined to notebooks/module-3/3.2_managing_messages.ipynb. Do not require changes to broad repository configuration, unrelated notebooks, dependency files, or package source directories.
Anti-leakage notes: Solver-facing materials must not mention benchmark, harness, Snorkel, Harbor, std-ts, agent, stage, attempt, retry, generated task, or evaluation context. Present the work as a normal notebook clarity and reliability improvement.
Approach: Have iterate create a regular coding prompt asking for a local preview of the existing SummarizationMiddleware example. The hidden tests should inspect the notebook JSON rather than running the whole notebook, so the work remains deterministic and grounded in the educational content shown in module 3.2.
Verification: Run python test_outputs.py from the task package root after applying a candidate patch. The test should report checks against notebooks/module-3/3.2_managing_messages.ipynb, including middleware configuration preservation, named message-list reuse, and deterministic preview output shape and counts.
Risk: The main risk is brittle notebook-cell matching. Mitigate by searching all code cells for function and variable definitions instead of depending on exact cell IDs, and by executing only isolated helper definitions so model-backed notebook cells are never called.
Path rules: task package is tasks/; mutable upstream checkout is tasks//environment/repo; benchmark package files are under tasks//task; verifier/container environment belongs under tasks//task/environment; regular and UI verifier logic belongs in task/tests/test_outputs.py and test.sh only runs that file through pytest.
Agent context
Task slug: notebook-message-summary-guardrails
Branch: task/notebook-message-summary-guardrails
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.