Install skills from databricks-agent-skills via databricks aitools - #562
Merged
Conversation
Collaborator
|
why not using the genie skill from das ? did you see this oe btw ? https://github.com/databricks-solutions/ai-dev-kit/pull/552/changes and https://github.com/databricks-solutions/ai-dev-kit/pull/553/changes ? |
Collaborator
|
should we make the skill folder we get from DAS empty as part of this PR to make everything coherent? |
Collaborator
Author
@QuentinAmbard It's typically going to create symbolic links to the folder that has the skills (.databricks/aitools/skills). Is that the folder you were referring to? |
Delegate the MCP venv build to databricks-mcp-server/setup.{sh,ps1} (new
canonical setup scripts) instead of inlining it in the installers. MCP now
defaults off with an experimental-style "Deprecated MCP Server" opt-in prompt
and a --mcp flag.
Also fixes the verified bash/PowerShell divergences from review:
- Claude global MCP path -> ~/.claude.json on Windows (was ~/.claude\mcp.json)
- $script:Profile_ typo in the Cursor-global message
- remove dead check_sdk_version (SDK check moved into setup scripts)
- add the Claude update-check SessionStart hook on Windows
- defer_loading parity; consolidate the three MCP-JSON writers into one
- env-var parity (DEVKIT_* + DEVKIT_BRANCH/AIDEVKIT_BRANCH alias)
- prompt_channel uses is_interactive; scope prompt reuses the shared selector
- generate the custom-skills picker from the live inventory
- PowerShell JSON writers init $existing and merge independent of the venv
Co-authored-by: Isaac
prompt_mcp_install ended with `[ "$selected" = "yes" ] && INSTALL_MCP=true`, which returns exit code 1 when the user picks "Do not install" (the default). Called bare under `set -e`, that aborted the installer before any skills were installed — with no error and no files written. Only triggered interactively; silent/non-interactive runs return earlier. Use an if-block so it returns 0. Co-authored-by: Isaac
The scope radio prompt's hints ("Install in current directory (.cursor/,
.claude/, .gemini/)") exceeded the terminal width once the arrow, dot, and
padded label were added, wrapping to two physical lines. The cursor-up redraw
in radio_select / Select-Radio counts logical lines, so each arrow press
stacked another copy of the menu. Shorten both installers' scope hints.
Co-authored-by: Isaac
Long hints could wrap past the terminal width, desyncing the cursor-relative redraw in the arrow-key selectors (each keypress stacked another copy of the menu). Make the selectors structurally immune regardless of content: - bash (checkbox_select, radio_select, skill-profile picker): disable line wrap (DECAWM, ESC[?7l) while drawing and restore it (ESC[?7h) on exit and in the EXIT trap, so the terminal clips overlong lines to one physical row. - PowerShell (Select-Checkbox, Select-Radio, skill-profile picker): truncate each hint to the remaining window width before printing. Co-authored-by: Isaac
…/kiro
`databricks aitools install` only fans out to a subset of agents — at project
scope it installs for Claude Code and Cursor and explicitly skips Copilot,
Codex, OpenCode, and Antigravity ("does not support project-scoped skills"),
while still populating the canonical store. Previously the installer only
symlinked the store into gemini/windsurf/kiro, so those skipped tools got no
skills.
Generalize delivery to symlink the canonical store into every selected tool's
skills dir, leaving entries aitools already created (e.g. Claude/Cursor) to
aitools. The temp-dir staging + copy fallback still applies when no
aitools-supported agent is selected. Mirrored in install.ps1; dropped the now
-unused UNSUPPORTED_AGENT_TOOLS / Get-UnsupportedSkillDirs.
Co-authored-by: Isaac
Drop the channel concept entirely: no release-channel prompt, no --experimental flag / DEVKIT_CHANNEL env var, no re-download-and-re-exec from the experimental branch, and no experimental banners in the summaries. The installer always uses the same path (latest release, or DEVKIT_BRANCH/AIDEVKIT_BRANCH). Installing a subset of databricks-agent-skills is done by picking a profile (e.g. --skills-profile data-engineer) instead of all. Unchanged: the separate `aitools install --experimental` flag (auto-added when a selected agent skill is experimental) and SKILLS_CHANNEL (raw-fetch dev knob). Co-authored-by: Isaac
Default true (current behavior: profiles and "all" include experimental databricks-agent-skills). Pass --experimental false (or DEVKIT_EXPERIMENTAL= false) to install only stable skills — experimental ones are filtered out of profile/"all" selections, and the aitools --experimental flag is then omitted. Explicit --skills requests are always honored as named, even experimental ones. Co-authored-by: Isaac
aitools prints "Skipped <agent>: does not support project-scoped skills" for agents it can't install at project scope (Copilot, Codex, OpenCode, Antigravity). The installer delivers to those tools itself via the canonical store, so the notice is just noise. Filter that specific line out of aitools' output on the non-silent path while keeping everything else (and real errors). Co-authored-by: Isaac
When MCP isn't selected (the default now), the summary no longer prints the MCP setup instructions that don't apply: "Enable MCP in Cursor", Copilot's "Configure Tools ... enable databricks", and "Restart Windsurf to pick up the databricks MCP server". Skills-only next-steps (Agent mode, launch/open the tool, try a prompt) still print, with correct step numbering. Co-authored-by: Isaac
Replace the "Try: List my SQL warehouses" line in the summary with a general statement to start prompting the AI assistant to interact with Databricks. Co-authored-by: Isaac
Deselecting everything in the custom picker left USER_SKILLS empty, so resolve_skills fell through to installing ALL skills. Force databricks-core into the selection (guaranteeing a non-empty explicit list) and warn when it's the only skill chosen. Mirrored in install.ps1. Co-authored-by: Isaac
Deselecting every option on the profile screen previously defaulted to installing all skills. Instead, treat an empty interactive selection as "custom" and show the individual skill picker (which preselects core and requires databricks-core). The silent/non-interactive default stays "all". Co-authored-by: Isaac
Add an optional "lock" field to the checkbox selectors: a locked item is always checked and can't be toggled off. Mark databricks-core locked (with a "(required)" hint) in the custom picker so users can see it's always installed and can't deselect it. Mirrored in install.ps1 (Locked item property). Co-authored-by: Isaac
An explicit --branch/-b (or DEVKIT_BRANCH / AIDEVKIT_BRANCH) now means "install that version's actual steps", not just clone it for the MCP source. When a ref is explicitly requested, the installer prints the command to run that branch's own installer (pinned with -b and a DEVKIT_BOOTSTRAPPED guard) and exits without installing. The guard suppresses the hand-off so the target installer proceeds. - install.ps1 gains a real --branch/-Branch flag (was env-var only). - Hand-off uses -b for bash and $env:AIDEVKIT_BRANCH for PowerShell, since old releases accept those (PowerShell never had a branch flag). - Default (no explicit ref) and bootstrapped runs are unchanged. Co-authored-by: Isaac
A silent run can't be handed off interactively, and exiting 0 without installing would look like success to automation. In silent mode the branch hand-off now writes the run-this-instead command to stderr and exits non-zero; interactive runs keep the friendly print + exit 0. Co-authored-by: Isaac
databricks-genie now comes from databricks-agent-skills (present in the live aitools inventory as of 0.2.5) instead of the bundled databricks-skills/ copy, and the APX raw-fetch source (databricks-app-apx from databricks-solutions/apx) is removed entirely. Skill sources are now just MLflow (raw fetch) + agent skills (aitools). - Drop LOCAL/bundled and APX source machinery: vars, buckets, install loops, ref resolution, lockfile/dry-run/list-skills/summary entries. - Add databricks-genie to the agent-skills experimental fallback. - Remove databricks-app-apx from the app-developer profile and pickers. - The repo clone is now only needed for the MCP server setup. Co-authored-by: Isaac
dustinvannoy-db
force-pushed
the
install-from-databricks-agent-skills
branch
from
June 23, 2026 16:46
1197fbc to
59158b3
Compare
Co-authored-by: Isaac (cherry picked from commit 8ed5aa7963487d2231956e9e6ed83acc77649351)
Co-authored-by: Isaac (cherry picked from commit bff71912fcfadc516de89e4bc743154f00af8192)
Co-authored-by: Isaac (cherry picked from commit aee0d5f5732a25684843a299b649f8a290f44cda)
Co-authored-by: Isaac (cherry picked from commit 334af81c477cd43dabc6ab6ec697d597d020ee50)
…kill validation - README: drop stale bundled-genie/APX skill sources and APX_REF/SKILLS_CHANNEL env vars; apply standalone MCP server + Builder App section wording; fix builder script names (start_local.sh); remove duplicate breaking-change note. - SECURITY.md: mark the Plugin Trust Model section deprecated. - validate_skills.py: discover skill dirs under databricks-skills/deprecated/. Co-authored-by: Isaac
Co-authored-by: Isaac
Reconcile the two migration approaches: keep this branch's 'databricks aitools' skill delegation (supersedes main's raw-GitHub-fetch of 3 skills) while grafting in main's new --uninstall feature (skills + MCP server + config + plugin sweep). Conflict resolutions: - install.sh / install.ps1: ours for all skill-sourcing/selection logic; kept main's auto-merged --uninstall machinery (UNINSTALL_SKILL_NAMES, run_uninstall/ Invoke-Uninstall, -y/--yes). Dropped main's raw-fetch vars and the removed release-channel concept; removed a duplicate --dry-run arg case. - README.md: kept main's 'major evolution coming' banner + Uninstall section AND our restructured skills sections; de-duplicated the rename note. - databricks-skills/README.md, install_skills.sh, builder skills_manager.py: ours (the deprecation work supersedes main's pre-deprecation text). - Accepted main's APX removal, VERSION 0.1.13, and minor wording changes. Co-authored-by: Isaac
…lers) On install, after scope is chosen, detect a prior AI Dev Kit install that predates the current databricks-aitools flow — real (non-symlink) skill dirs under known names with no .databricks/aitools/skills/.state.json store, and/or the Claude Code plugin — for the target scope. When found, warn and (interactively) offer to run the full scope-aware uninstall first, then continue on a clean slate; non-interactive/silent only warns. Skills already managed by aitools (a CLI upgrade or a prior run of this installer) are detected via the aitools store and never flagged, so upgrades are not interrupted. install.sh runs the uninstall in a subshell (run_uninstall exits); install.ps1 calls Invoke-Uninstall directly (it returns) with AssumeYes forced then restored. Co-authored-by: Isaac
…plugin The retired plugin is specifically this repo's databricks-ai-dev-kit marketplace plugin. It is NOT a move away from plugins: databricks aitools install (CLI v1.6.0) installs the official databricks plugin through each agent's own plugin CLI for Claude Code, Codex, and GitHub Copilot, and writes raw skill files for Cursor, OpenCode, and Antigravity (the skills-only flag forces files everywhere). Updated DEPRECATED.md, the README Skills section, and the plugin.json/marketplace.json descriptions to say so. Co-authored-by: Isaac
…1 skills-only (#585) * Extract MCP server install into its own installer; make install.sh/ps1 skills-only Add databricks-mcp-server/mcp_install.sh and mcp_install.ps1: standalone MCP installers that delegate the venv build to setup.sh/setup.ps1 (via --venv-dir / -VenvDir) and register the server with all nine supported clients (Claude Code, Cursor, GitHub Copilot, OpenAI Codex, Gemini CLI, Antigravity, Windsurf, OpenCode, Kiro). Each client keeps its exact config shape/root key/format, and every client — including Codex — gets an "env": {"DATABRICKS_CONFIG_PROFILE"} block. No host or token is ever written. Safe merge with .bak backup and a symmetric --uninstall / -Uninstall. Remove the MCP option from install.sh and install.ps1 entirely: drop the --mcp/--mcp-only/--mcp-path flags and DEVKIT_INSTALL_MCP/DEVKIT_MCP_PATH env, the INSTALL_MCP gating, and all MCP-only helpers (setup_mcp, clone_repo, the write_mcp_* / Write-Mcp* / Write-ClaudeHook config writers). These installers now set up skills only and point users to the new MCP installer. The uninstall path keeps its MCP cleanup so registrations from older install.sh --mcp runs can still be removed. setup.sh and setup.ps1 are untouched. Update databricks-mcp-server/README.md Step 3 to recommend mcp_install.sh/.ps1 and add the DATABRICKS_CONFIG_PROFILE env block to the manual snippet. Co-authored-by: omnigent <noreply@omnigent.ai> * Handle removed MCP flags/env gracefully instead of hard-failing Both installers now detect the now-removed MCP options and print a short, actionable deprecation notice to stderr pointing at databricks-mcp-server/mcp_install.sh (or mcp_install.ps1 on Windows): - --mcp and --mcp-path <x>, plus DEVKIT_INSTALL_MCP=true: warn, then continue the normal skills-only install. --mcp-path consumes its value so arg-parsing doesn't choke (the bash side guards the shift so a bare --mcp-path can't trip set -e). - --mcp-only: no non-MCP work to do, so print the pointer and exit 0 cleanly. Previously these aborted with an error. setup.sh / setup.ps1 remain unchanged. Co-authored-by: omnigent <noreply@omnigent.ai> * Fix set -e abort, PS profile flag, JSON no-clobber parity, TOML anchor Address cross-vendor review of PR #585: - B1 (blocking): mcp_install.sh guarded the top-level SCOPE_EXPLICIT assignment with a full `if` so a false test can't abort the script under `set -e` on older bash (macOS /bin/bash 3.2). Audited the rest of the script; line 58 already had a trailing `|| SILENT=false`, and no other top-level bare tests can abort. Verified `--help` runs cleanly with DEVKIT_SCOPE unset on both bash 3.2 and 5.2. - N3: removed the dead ProfileProvided assignment in mcp_install.ps1 (it read the raw param, not the resolved value, and was never used anywhere). - N2: gave mcp_install.ps1 the same no-clobber behavior as bash - Write-McpJsonConfig and Write-OpenCodeJson now refuse to merge into an existing config when the venv python is absent and warn "add manually", instead of merging regardless. Chose to keep the SAFE behavior in both: bash relies on the venv python as its JSON parser and cannot merge without it, and a config pointing at a missing interpreter is useless. New files are still written from the template in both. - S2: anchored the Codex TOML idempotency/detection checks to the table header in both installers so a match inside a comment or value can't be mistaken for an existing registration. This aligns detection with the removal blocks (which already anchored), so uninstall still matches. setup.sh / setup.ps1 remain unchanged. Co-authored-by: omnigent <noreply@omnigent.ai> --------- Co-authored-by: omnigent <noreply@omnigent.ai>
Default the MCP server virtual environment to the repository root instead of databricks-mcp-server/.venv while preserving explicit --venv-dir / -VenvDir overrides and mcp_install pass-through behavior. Co-authored-by: omnigent <noreply@omnigent.ai>
…ope docs Clean up the MCP server README to match the actual scripts and server: - Regenerate the "Available Tools" table from the 44 registered @mcp.tool functions. The old table listed ~30 tool names that no longer exist (the API was consolidated into manage_* action-dispatch verbs); add an Actions column and cover all product areas (Unity Catalog, Vector Search, Lakebase, Apps, Metric Views, etc.) that were previously undocumented. - Fix step numbering (there was no Step 3): Authenticate is now Step 3, Smoke test Step 4, and the "see Step 3/4" cross-references are corrected. - Remove the duplicated dangling code fence under the installer command. - Correct the `claude mcp add-json` snippet: it writes to ~/.claude.json, not the project .mcp.json, unless `-s project` is passed; also restore defer_loading:true. - Document install scope: add a client -> config-file table for project vs. global scope (Claude Code global = ~/.claude.json), and note the repo must stay cloned since configs point at run_server.py by absolute path. - Note uv is required on macOS/Linux (with --skip-venv escape hatch) vs the python -m venv fallback on Windows. - Fix the stale "run install.sh and choose the skills option" line (install.sh is skills-only now). - Move the architecture diagram out to ARCHITECTURE.md and link to it. - Fix the smoke-test suggestion to use tools that actually exist. Co-authored-by: omnigent <noreply@omnigent.ai>
The tools-core box listed a 'pipelines/' package that does not exist (the real package is spark_declarative_pipelines/) and omitted the largest package, unity_catalog/. Replace the package row with real package names and a trailing ellipsis (consistent with the diagram's high-level-sketch disclaimer).
Pure cleanup — no behavior change. Each item below was verified to have zero reachable call sites or reads: - install.sh: drop `deprecation_notice()`. The "skills are moving" announcement has shipped and is being retired; the function had no call sites. - install.ps1: drop `Show-DeprecationNotice` for the same reason. - install.ps1: drop the `$VenvDir` / `$VenvPython` block. The venv now lives in databricks-mcp-server/mcp_install.ps1, and Invoke-Uninstall edits JSON via ConvertFrom-Json/ConvertTo-Json rather than shelling out to a Python interpreter, so neither variable is read anywhere. (The bash `VENV_PYTHON` is still live and is left alone.) - install.ps1: drop the vestigial `$line = " "` assignment inside the `$drawCheckbox` scriptblock — assigned, never read. - install.ps1: drop the unreachable `$i++` after `Write-Err` in the arg-parse `default` arm; `Write-Err` ends in `exit 1`. The MCP deprecation shims, the no-op `--skills-only` arm, the AgentBExcluded knob, and the uninstall MCP-cleanup helpers are all reachable and left untouched. Co-authored-by: omnigent <noreply@omnigent.ai>
dustinvannoy-db
marked this pull request as ready for review
July 27, 2026 01:31
dustinvannoy-db
requested review from
QuentinAmbard,
calreynolds,
jacksandom,
malcolndandaro and
scottDBX1886
July 27, 2026 01:31
Collaborator
Author
|
@QuentinAmbard @malcolndandaro This one is ready for review, but we don't want to merge until we are ready to announce (late Monday or early Tuesday). I tested many variations of install but would love more eyes on README.md, install.sh, install.ps1, and databricks-mcp-server/mcp_install.sh. |
scottDBX1886
left a comment
scottDBX1886
left a comment
Collaborator
There was a problem hiding this comment.
reviewed install scripts and README (root). Everything looks good from my review/agent review. No blocking issues found from my side.
calreynolds
previously approved these changes
Jul 27, 2026
malcolndandaro
previously approved these changes
Jul 27, 2026
malcolndandaro
left a comment
malcolndandaro
left a comment
Collaborator
There was a problem hiding this comment.
Tested manually on Windows + Copilot (VS Code). Everything worked.
Readme looks good.
LGTM
dustinvannoy-db
requested review from
calreynolds,
malcolndandaro and
scottDBX1886
July 27, 2026 21:10
…ks-agent-skills # Conflicts: # databricks-builder-app/README.md # databricks-builder-app/scripts/_integration-example/setup.sh # databricks-builder-app/scripts/deploy.sh # databricks-builder-app/scripts/start_local.sh # databricks-builder-app/server/services/skills_manager.py
scottDBX1886
approved these changes
Jul 27, 2026
malcolndandaro
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks both installers (
install.sh/install.ps1) so that most Databricks skills are sourced from databricks/databricks-agent-skills by delegating todatabricks aitools install(Databricks CLI v1.0.0+), instead of bundling them in this repo. Implements.local/implementation_plan_INSTALL_DAS.md.New skill source map
databricks-execution-computeexcluded from defaults)databricks aitools install --scope … --agents … --skills …databricks-geniemain(MLFLOW_REFoverride)databricks-app-apxAPX_REFoverride)The agent-skills inventory is discovered live via
databricks aitools list -o json(so new upstream skills flow into theallprofile automatically), with a hardcoded v0.2.3 snapshot as offline fallback.Key behaviors
aitoolsowns the agent skills after install (aitools list|update|uninstall). The installer manifest only tracks skills it installs directly. Profile changes uninstall dropped agent skills viaaitools uninstall.aitoolswill not overwrite a pre-existing real directory with the same skill name (verified against CLI v1.0.0), so the manifest cleanup removes old bundled copies of now-agent-managed names before delegating. Upgrades from older installs were validated end to end..databricks/aitools/skills); if no aitools-supported tool is selected, a throwaway temp-dir install stages the files for copying so no unselected agent dirs are touched. Windows falls back to copying when symlink creation is not permitted.databricks-bundles→databricks-dabs,databricks-spark-declarative-pipelines→databricks-pipelines;databricks-configreplaced bydatabricks-core,databricks-lakebase-autoscale/-provisionedbydatabricks-lakebase. Explicit--skillsrequests for old names are migrated with a warning; unknown names fail loudly.latestresolves to the highest stable semver tag viagit ls-remote(no GitHub API rate limits;INCLUDE_PRERELEASES=1to allow-rctags). New env vars:APX_REF,MLFLOW_REF,SKILLS_CHANNEL,INCLUDE_PRERELEASES.--dry-runprints resolved refs, the exact aitools command, and the delivery plan, then exits without changes. Askills.lockin the scope-local state dir records resolved refs/SHAs and the aitools release.--list-skills, the interactive custom picker, and help text are reworked around the new names/sources.Deviations from the plan (evidence-driven)
copilot, notgithub-copilot(verified against the CLI).git ls-remoteinstead of the GitHub tags API after hitting 403 rate limits during testing.databricks-vector-searchis now stable anddatabricks-lakeflow-connectexists; handled by the dynamic discovery.Testing
install.sh: full silent project installs in sandbox dirs (claude+gemini and gemini-only), profile-change cleanup incl.aitools uninstall, stale-copy upgrade replacement, rename migration, unknown-skill failure,--dry-run,--list-skillsagainst the live 0.2.3 inventory, apx tag resolution to v0.3.8, lockfile contents.install.ps1: parses clean (PowerShell 7.5.4 parser),--list-skillsand silent--dry-runproduce output identical to bash on macOS pwsh. Not yet executed on real Windows — the symlink-privilege fallback and winget upgrade path need a Windows smoke test before merge.Follow-ups (out of scope)
databricks-skills/.databricks-builder-app/README.mdanddatabricks-mcp-server/README.md.This pull request and its description were written by Isaac.