Skip to content

About

ExperimentsAgent: designs, runs and explains AutoML studies in Zetta

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

agent-experiments

The ExperimentsAgent: designs AutoML studies for a dataset, runs them in Zetta after the person confirms, watches trials and their training metrics live, stops diverging trials, explains the result, and promotes the best trial after confirmation. Built on agent-runtime.

Current release: v0.3.0.

Its image is published as oramasearch/agent-experiments on Docker Hub.

The Dockerfile pins its bases by multi-platform index digest (lake#1003): python:3.12-slim@sha256:5e2bdb1b… for both stages and ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85a…. Moving one is a reviewed change of its digest.

Service

The Python service exposes POST /v1/turn and GET /healthz on port 8080; /healthz reports the package version and prompt_version. It can also receive agent_turn tasks of kinds experiment, experiment_status, and experiment_explain from lake's agent bus with AGENT_BUS_TOKEN and AGENT_BUS_AGENT. A person's delegated lake credential is separate from the service and bus credentials.

Development uses Python 3.12 and uv:

uv sync
uv run pytest

See the documentation index for the service contract, study workflow and release notes.

The model is reached through lake (lake#248, lake docs/api/model-gateway.md): every call goes to <LAKE_URL>/openai/v1 with the turn's own lake_token as the API key (model.py), so lake counts its tokens in the turn's workspace and enforces the account's model-token limit. The service holds no OPENAI_API_KEY; only the offline evals, which run for no workspace, call OpenAI directly.

A turn's credential goes only to Lake at this service's own LAKE_URL. The model address is always LAKE_URL. A turn's lake_url, the workspace path Lake hands the agent (<LAKE_URL>/w/<workspace>/workspaces/<customer>/zetta), is accepted only on LAKE_URL's origin (scheme, host and port) under its path, with no user info, query or fragment; any other address refuses the turn before a request is made. Without a lake_url, the turn uses LAKE_URL itself. With LAKE_URL unset, every turn is refused.

Conversation history is bound to the workspace and bounded (#35). The pod keeps a design conversation's messages in memory keyed by the turn's customer, workspace and conversation id, so a turn from another workspace with the same id starts empty. At most 256 conversations are kept, an idle one expires after an hour, and each keeps at most 60 messages and 200,000 characters after its system prompt (service operation).

Study and confirmation contract

An initial experiment turn reads dataset, project, pipeline, feature, methodology, and optional solution facts through lake and returns result.kind: proposal with confirm: true. It defaults a single relevant project and an out-of-time validation split when dataset time evidence supports them. A project without a suitable pipeline produces a dataset study instead of a placeholder pipeline. Every external reference is checked before review and checked again after confirmation. The search space comes from the methodology's published study_search_space, and the complete study is preflighted with zetta's validate_study (the same checks as create_study, storing nothing); a rejection is fed back to the model with its path and reason before a person sees it. The initial turn performs no action. A follow-up turn must carry the reviewed object in context.confirmed_proposal; only then does the agent call create_study.

experiment_status tails study-events.<study> and run-metrics.<run>, reports budget and trial progress, and stops trials with non-finite loss, exploding gradient norm, or a sustained overfitting gap. experiment_explain reports the best trial, parameter importance, next search, and caveats. Promotion requires the reviewed object in context.confirmed_promotion; plain text never authorizes promote_trial.

Verification and evals

uv run pytest covers the service, bus, proposal schema, confirmation gates, exact lake capabilities, event cursors, divergence rules, analysis, and promotion. Run uv run experiments-agent-evals for the known-good search-space, non-finite-loss stop, and promotion-without-confirmation cases against deterministic fake lake facts.

License

MIT, see LICENSE. Third-party components and their licenses are in THIRD_PARTY_LICENSES. Contributions are welcome under the Contributor License Agreement; see CONTRIBUTING.md.

About

ExperimentsAgent: designs, runs and explains AutoML studies in Zetta

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages