advanced topics

11 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Managed Deep Agents evals are Harbor tasks

Managed Deep Agents evals are Harbor tasks. Use a coding agent with the eval-engineering skill to inspect the project, draft a Task Spec for review, and write complete tasks under evals/. Managed Deep Agents initializes the Harbor workspace, and Harbor runs the managed agent against each task in an isolated environment and records the result.

Prerequisites for evaluating Managed Deep Agents

Before evaluating Managed Deep Agents, you must have: a Managed Deep Agents project created with mda init or an existing project with an agent entry; uv installed, which runs the pinned Harbor version and plugins; Docker installed, which Harbor uses for task environments; and a coding agent that supports Agent Skills.

Add eval-engineering skill to project

To add the eval-engineering skill to the current project, run: uvx --from npx-skills skills add langchain-ai/langchain-skills --skill eval-engineering --yes for Python, or npx skills add langchain-ai/langchain-skills --skill eval-engineering --yes for JavaScript.

Install Deep Agents Code (dcode)

To use Deep Agents Code (dcode), install it with: curl -LsSf https://langch.in/dcode | bash. Then see the Deep Agents Code quickstart for provider setup and interactive use.

Initialize eval workspace with mda evals init

From the project root, initialize the eval workspace by running: uv run mda evals init -i for Python, or npx mda evals init -i for npm, pnpm exec mda evals init -i for pnpm, or bunx mda evals init -i for bun. The interactive handoff lists detected coding agents including Deep Agents Code, Claude Code, Codex, and Cursor. Selecting an agent starts that agent in the project directory and runs the eval-engineering prompt.

Eval workspace initialization creates specific directories and files

Initialization of the eval workspace creates: evals/harbor-job.json (user-owned, preserved on later runs), evals/ directory, and .mda/evals/ directory containing runtime.json and harbor-adapter/ (generated files). evals/harbor-job.json is user-owned and Managed Deep Agents writes it only when it is missing, so later edits are preserved.

Harbor task structure and layout

Each direct child of evals/ that contains an instruction and tests is a Harbor task with the following structure: Task.md (human-reviewed spec), instruction.md (tells the agent what to do), task.toml (task configuration), environment/Dockerfile (task environment), and tests/test.sh plus optional verifier files. Harbor builds the task environment, runs the managed agent, and then runs tests/test.sh. The verifier writes a numeric reward to /logs/verifier/reward.txt or numeric metrics to /logs/verifier/reward.json.

Run evals command with Harbor and LangSmith plugins

On macOS or Linux, the eval run command has the form: HARBOR_LANGSMITH_DATASET=mda-my-agent-evals PYTHONPATH=.mda/evals/harbor-adapter uv run --env-file .env --python 3.12 --with 'harbor[langsmith]==0.21.0' harbor run --config evals/harbor-job.json --yes --plugin mda_harbor.job_plugin:MDAJobPlugin --plugin mda_harbor.langsmith_plugin:LangSmithPlugin. Replace my-agent with the project directory name. The generated command fills in the name and uses PowerShell syntax on Windows. Re-running after editing the agent picks up project changes.

View Harbor results

Open the Harbor results by running: uv run --python 3.12 --with 'harbor[langsmith]==0.21.0' harbor view .mda/evals/jobs. Review failed trials with your coding agent. Update the task or verifier when the eval does not measure intended behavior, or update the managed agent when the eval exposes a product failure, then run the Harbor command again.

Edit Harbor job configuration

Edit evals/harbor-job.json to change datasets, attempts, concurrency, environment settings, or agent environment variables. Managed Deep Agents preserves the file when you run mda evals init again.

Record Harbor runs in LangSmith

When LANGSMITH_API_KEY is available, the LangSmith plugin records the Harbor runs in the dataset named by HARBOR_LANGSMITH_DATASET environment variable.

Give your agent this brain