Skip to content

experiment: document and reproduce shared Gemara policy gates for Fullsend auto-merge #64

Closed

Description

@ascerra

Companion research issue

Fullsend research findings and problem-space documentation. This experiment supplies the public reproduction and results record; the companion issue connects those findings to Fullsend’s existing research questions.

Objective

Create a public, self-contained experiment that teaches what we built and learned while using Gemara to assess Fullsend PRs. Capture the completed lab observations, provide safe offline reproductions, and clearly separate shared policy, repository bindings, evidence collection, policy assessment and the App/SCM merge action.

This is a retrospective experiment publication with reproducible synthetic scenarios. It is not a request to deploy a merge writer, run a model, contact a private telemetry service, change another repository or repeat a live merge. All implementation changes for this issue stay in fullsend-ai/experiments.

Repository conventions and proposed deliverables

Follow AGENTS.md and the experiment template. At this snapshot, 0028 is the next unused numbered experiment; recheck before creating it.

Suggested directory: NNNN-gemara-auto-merge/.

  • README.md: hypothesis, approach, results summary, conclusion, one-command offline walkthrough and diagrams; required frontmatter and matching title/H1.
  • RESULTS.md: historical scenarios and their evidence; failed attempts and lessons; explicit limits and unanswered questions.
  • docs/OWNER-WALKTHROUGH.md: readable owner flow from policy selection to issue → PR → review → assessment → queue → merge, with source links at each step.
  • docs/OWNERSHIP-AND-MAINTENANCE.md: upstream/shared/consumer/forge responsibilities, version pins, update procedure and qualified LOC inventory.
  • evidence/sources.json: public source register with immutable revisions, public run/PR URLs, claim IDs, artifact names/IDs when available, observation dates and evidence status.
  • fixtures/: small, invented, deterministic GitHub/GitLab candidate observations, judge aggregates and recorded GitHub check/status responses. Label every fixture synthetic. Do not copy raw live trace/artifact payloads.
  • scripts/ and focused tests: a minimal offline scenario runner, importing the pinned public shared module where useful. Do not copy the whole auto-merge repository, vendored Fullsend binary, videos, credentials or live writer.
  • Update the root experiment index and any necessary scoped checks. Include dependency and content notices if copying upstream licensed material.

Use status Concluded if publication and offline reproduction are complete and no additional runs are planned. Do not use Merged: this records a lab result, not adoption into Fullsend. The broader policy-engine choice remains undecided.

Relationship to existing Fullsend research

This experiment supplies a concrete observation for the existing problem docs; it does not resolve their architecture questions. Cross-link these public areas in the experiment and link the companion Fullsend research-update issue above:

Research questions and hypotheses

  1. Can one shared policy/evaluator express owner requirements for multiple consumers while keeping forge identity, trusted evidence and mutation local to each consumer?
  2. Can a separate quality condition hold an otherwise passing PR, and can its threshold change independently of the measurement?
  3. Can selecting a maintained OpenSSF requirement add useful behavior to an existing policy, and how much custom implementation remains necessary?
  4. Can assessment output feed an existing protected GitHub merge path while preserving the distinction between policy eligibility and owner-authorized mutation?
  5. What user experience and maintenance cost does this add compared with the hoped-for simple policy-selection experience? What did this experiment not establish about Gemara's suitability?

Architecture to explain

flowchart TD
    O[Owner selects trusted policy and configuration] --> B[Consumer binding and current revisions]
    F[Fullsend Review and forge checks] --> E[Consumer evidence adapter]
    M[External judge assessments in MLflow] --> C[Read-only bounded aggregate collector]
    C --> Q[Credential-free quality assessment]
    U[OpenSSF maintained requirement] --> P[Our assessment mapping and GitHub check adapter]
    S[Pinned shared catalog, policies and evaluator] --> G[Native Gemara assessments and logs]
    E --> G
    B --> G
    P --> G
    Q --> D[Our AND-composed eligibility decision]
    G --> D
    D --> H[Hold with reasons or eligible]
    H --> W[Separate trusted authorizer and scoped App writer]
    W --> K[GitHub protected queue and combined-revision checks]
    K --> R[GitHub performs merge]
Loading

The diagram is a dependency/authority map, not proof that Gemara schedules collectors or runs a judge. Spell out that Gemara does not create the MLflow judge, authenticate GitHub evidence, mint App tokens, grant owner authority or merge PRs.

Division of responsibilities

  • Gemara upstream: schemas/model and Go SDK assessment/evaluation machinery.
  • OpenSSF upstream: OSPS requirement content and applicability. This is maintained content, not a ready-made GitHub scanner.
  • Our shared public module: common catalogs/plans, deterministic evaluators, judge aggregate checks, CLI, normalized types, synthetic examples and collector.
  • Consumer repo: trusted source pins, forge identity, current PR/base/merge-group subject, evidence publishers/collectors, owner scope, quality threshold/window/freshness, selected optional controls and credential boundary.
  • Existing writer/forge: revalidation, short-lived repository-scoped write identity, protected enrollment, combined-revision CI/assessment and actual merge.

Reusing policy does not mean sharing write credentials, trusting a PR's policy changes for its own authorization, or treating a JSON digest as an authenticated source. The default public reproduction is read-only/offline.

Historical scenarios to capture accurately

A. Basic Fullsend and review-evidence path

Briefly describe the baseline work that made normal Fullsend review output consumable: completed approval, unresolved findings, risk assessment and exact current revision/configuration binding. Link the public PR #50 and PR #52 as historical preparatory changes. Do not imply every preparatory PR was a functional merge demonstration. Explain why current-head CI and a completed AI review are separate evidence sources.

B. Reusable common policy and the missing-risk hold

List the shared candidate controls: OWNER-SCOPE, REVIEW-CLEAR, CHECKS-PASS, EVIDENCE-CURRENT, PROTECTIONS-ACTIVE. Review-clear also requires the configured completed review/risk evidence; a clear-looking review with no required risk assessment must not pass.

Use the public shared GitHub/GitLab bindings and synthetic observations to show that the policy is reusable while evidence adapters remain forge-specific. The original GitLab consumer was expected to hold because its review output lacked the required risk assessment. Describe the live deployment only generically as private historical context; do not publish its URL, project name, infrastructure identifiers, logs or credentials. Publicly reproducible evidence for this case is the synthetic fixture. Do not claim a public live GitLab merge or identical GitHub/GitLab enforcement infrastructure.

Record that runner restrictions complicated isolated shared-code execution; controls around a pinned dependency and credential separation are consumer integration concerns. Do not advertise executing shared code beside reader or writer credentials as safe merely because we control the shared repo.

C. Additional MLflow quality requirement: 99% holds

The external scorer is named review_correctly_approved. Its historical observation had 238 passing judgments / 254 scored approvals = 93.7008…%. At a 99% threshold, the quality condition failed even though the five common candidate requirements passed. Our composed decision held, and the writer skipped protected queue enrollment. Link PR #53 and the historical public assessment run/artifact.

For offline reproduction, construct a synthetic aggregate with the same counts and a clearly synthetic source identity. Never claim this invented fixture is an authenticated export of the private MLflow service. Keep the collector's public implementation link; do not link its private deployment or the internal judge implementation repository.

Explain what the collector does: read-only metadata aggregation, fixed scorer/source/experiment/window, bounded pagination and population checks, binary-score handling, freshness/sample requirements and aggregate output. No model is invoked by Gemara or this offline reproduction.

D. Owner changes quality threshold to 90%: eligibility and protected merge

Use a separate configuration/fixture with 9000 basis points, preserving the 9900 example so both cases remain inspectable. With the same 238/254 observation, the quality condition passes. This does not improve review quality or override another failing requirement.

Link merged PR #56 as the earlier 90% case. PR #54 is not the successful merged example and must not be described as such. For the complete later trial, show issue #57 → PR #58 and the public source chain below.

E. Add one maintained OpenSSF requirement

Teach the complete mapping:

  1. Select OSPS-QA-03.01 from pinned OpenSSF OSPS-QA.yaml; retain original content and source/license attribution.
  2. Our catalog wrapper selects the parent control/requirement; our policy maps it to assessment method all-checks-passed.
  3. Our GitHub adapter examines exact-revision check runs and commit statuses, including optional observed pass/fail checks. Required-check enforcement remains independently in place.
  4. Our native step supplies the observation to the Gemara SDK; the native evaluation log identifies the upstream requirement and result.
  5. Our composed decision and existing writer require that result along with the other conditions.

Show an offline case where required checks pass but an optional automated check fails: the selected requirement must hold. Also test pending checks, absent evidence, incomplete API data and the current queue evaluator's narrowly identified self-check exclusion. Describe neutral/skipped/rerun semantics from the implementation; do not simplify these into “every GitHub check must say success.” The lab supports no automatic manual-bypass exception and is stricter than the upstream manual-bypass allowance. Passing this requirement does not mean meeting the whole baseline or every maturity-level control.

F. Complete protected merge and post-trial cleanup

PR #58 was merged on 2026-10-23 at 13:03:51 UTC, commit fee00797c19a07a2e3bdd83ce7fd1dbc917087f1.

Walk through the actual user-visible evidence in order:

  1. Issue chore: update fullsend per-repo installation #57 describes the requested documentation behavior; PR experiment: mutation testing for eval coverage (muteval on 0016) #58 implements it.
  2. Fullsend Review completed for head 14499514434b02c4b2563cd0e50aec225b77fa86 against trusted configuration 5f4f8f39ff196d815866396e6f05ebe6aff6e7a9: approval, no unresolved findings, low risk.
  3. Shared candidate conditions, the 90% quality requirement and selected OpenSSF requirement passed; native artifacts record the assessments.
  4. The separate trusted App writer revalidated and requested protected queue enrollment. The forge timeline records gemara-auto-merge-pilot-lab[bot] adding the PR to the queue at 12:58:50 UTC; token revocation was observed afterward.
  5. Queue assessment and CI ran on cumulative revision fee00797…, rather than treating the reviewed PR head as equivalent to the merged result. The selected upstream requirement passed at both enrollment and queue validation.
  6. GitHub performed the merge. The bounded lab authority allowed the trial; the broader production-readiness assessment still held.
  7. Cleanup commit 4c86073e… returned configuration to shadow and disabled grants; final CI passed. The historical protection comparison recorded exact restoration. Do not claim a live read-only rules page proves its historical state.

Disclose that earlier setup deployments used separately approved temporary admin bypasses with restoration. The demonstration PR used the protected queue. Do not include bypass instructions or make them prerequisites for reproduction.

Owner experience and configuration lesson

Show the actual experimental configuration:

auto_merge:
  mode: shadow
  policy: fullsend-default
  controls:
    - OSPS-QA-03.01

Explain where it lives in .fullsend/config.yaml, or under the repository's config: / inherited defaults.config in a managed repos.yaml. The lab's locally modified Fullsend build materialized these settings and supported an effective-config inspection command. off overrides inherited enablement; an empty controls list clears inherited optional selections.

This schema/build support was local experimental work carried as a vendored binary and source patch, not a released Fullsend feature. repos.yaml handling was locally tested; do not claim a live managed GitLab installation exercised it. enforce alone cannot mint authority or renew an expired grant. Adding an arbitrary upstream control is not plug-and-play until a trusted implementation/evidence adapter exists.

The owner walkthrough should point to the exact file, explain why it exists and who maintains it, then show the corresponding PR/run/artifact. Prefer a compact Mermaid flow and optionally a self-contained HTML explainer with clickable sources. A video or screenshot is supplementary, not a substitute for text and public evidence links. No external narration service or third-party telemetry transfer is required by this issue.

Failures and lessons worth retaining

  • An unsigned contribution correctly drew a DCO finding; the branch was corrected and fresh checks/review were run. Do not present the rejected revision as approved.
  • A mismatch between the vendored runtime pin and dispatcher provenance needed separate identities/pins; success required correcting provenance, not weakening evidence validation.
  • Too much experiment authorization/history mixed into a demo PR created review friction. Keep setup, owner scope and the demonstration issue understandable and distinct.
  • Passing a workflow job can mean “observation completed,” while the policy inside its artifact still reports hold. Teach readers to inspect the decision/result, not only the green job icon.
  • Schema-valid output, successful collection, candidate eligibility, owner authorization, queue enrollment, merge-group success, actual merge and production qualification are different outcomes.

Offline reproduction and acceptance tests

Provide a single documented command that runs without secrets, GitHub write access, MLflow access or LLM calls. A pinned dependency download may be needed during setup; after setup, scenario execution must work offline. Use the existing shared CLI/library rather than reimplementing its policy semantics in a parallel engine.

Required assertions:

  • Clean candidate + 238/254 at 99% → quality Failed and composed hold.
  • Same candidate/aggregate at 90% → quality Passed and candidate eligible, with no mutation or implicit authorization.
  • Exactly at a configured boundary passes; one sample below fails using integer arithmetic, not rounded dashboard percentages.
  • Missing risk → hold, for the synthetic GitLab consumer despite approval/CI appearing clear.
  • Required CI green + observed optional check failed → selected OSPS requirement holds; all relevant successful checks → passes.
  • Missing/stale/insufficient/malformed judge evidence → Unknown/hold, not an empty-population 100% pass.
  • Candidate/current-head mismatch or untrusted evidence → hold.
  • Shadow mode and absent/expired authority cannot cause a writer action even when assessments pass. Test through mocks/pure decision paths; do not create a live writer.
  • Candidate and merge-group subjects remain distinct in synthetic fixtures and historical evidence.
  • Repeated execution produces deterministic results at an explicit fixed evaluation time.

Do not represent mocked writer assertions as a second live protected-merge test. Do not fabricate results for future scenarios: run them and label new local results separately from historical hosted observations.

Public-safety and evidence requirements

Use only public code, public forge metadata and newly authored synthetic fixtures. The source register below is the starting allow-list; reverify availability and licensing when implementing.

  • Never include live service URLs, internal GitLab details, infrastructure/IP-derived hostnames, usernames/passwords, App private keys/tokens, reader credentials, internal repository links, raw traces or customer data.
  • Do not copy local narrated demos, screenshots, archives or downloaded artifact bundles wholesale. Publicly hosted artifacts may still contain sensitive or unnecessary data; their location is not clearance to republish every field.
  • If archiving a historical public result, produce a reviewed allow-listed extract of necessary identifiers/counts/results, record its public source, extraction date and digest, and preserve its historical status. Fetching private evidence or expanding public disclosure requires a separate decision; neither is needed for this task.
  • Preserve existing upstream license notices for any copied code/content. Prefer pinned imports and concise extracts to vendoring entire implementations.
  • Scan the actual proposed publication files for secrets/internal identifiers and inspect the resulting diff. Scope tooling to the new experiment/index; do not rewrite unrelated experiments.
  • Actions artifact links can expire or require login. Include enough reviewed public metadata plus synthetic replay to keep the explanation useful without promising anonymous artifact downloads or perpetual availability.

Value assessment and unresolved questions

Document the measured owned footprint rather than claiming effortless integration: shared 1,726 source/tooling / 960 test / 619 policy-config-CI lines; auto-merge 7,215 / 5,157 / 2,649 respectively; local Fullsend +190/−6 source and +170 test lines. OSPS-specific dedicated files total 454 physical lines (165 adapter, 132 policy/pin, 157 tests), excluding reused generic machinery. Qualify comments/blanks, retained utilities, dependencies and historical revisions; these are neither minimum adoption cost nor engineering time saved. Put detailed method/inventory in the maintenance doc rather than dominating the owner walkthrough.

Assess observed benefits: common artifact vocabulary, reusable shared evaluation, upstream requirement provenance, explicit missing/failed results, and traceable evidence-to-decision reporting. Also assess cost: custom trusted adapters, schema/version management, source authentication, protected execution, queue reconciliation and user comprehension.

Remain explicit about what is unproved: statistical validity of the threshold, scorer correctness, independent human agreement, scored-population coverage, agent/rubric-version cohort isolation, long-term availability, production rollout safety, comparative superiority over simpler native policy structures, and live GitLab merging. The historical 93.70% is a pass rate among scored approvals, not overall review accuracy or a probability that the next PR is safe.

Definition of done

  • Numbered, indexed experiment with correct metadata, a source-backed results narrative and an honest conclusion.
  • Owner flow and shared-versus-consumer maintenance responsibilities are understandable without reading this issue or the private chat.
  • Offline reproductions and meaningful boundary/negative tests pass; expected hold cases are explained as successful demonstrations of policy behavior.
  • Public source register distinguishes historical hosted evidence, immutable implementation sources, synthetic cases, proposals and unknowns.
  • Exact actual writer/queue/merge evidence is shown; Gemara is never credited with performing the merge.
  • No live mutation capability, production default, private resource dependency or secret is introduced.
  • Scoped experiment/frontmatter/index/code checks and publication review pass. Follow repository contribution/DCO rules; this issue itself does not authorize cross-repository changes or a new live test.

Public source register

Sources below were checked on 2026-10-24. Source-code links are immutable snapshots; issue, PR and run pages describe historical observations and can change independently. GitHub Actions artifact downloads may require GitHub authentication and expire. An artifact link is a locator, not a permanent evidence archive.

Upstream building blocks

Shared policy and consumer implementation

Hosted evidence

Related Fullsend work — not adoption evidence

Activity

  1. fullsend-ai-triage commented on Oct 24, 2026

    @fullsend-ai-triage

    🤖 Finished Triage · ✅ Success · Started 4:48 PM UTC · Completed 4:51 PM UTC

    Commit: 151b860 · View workflow run →

    Runtime: claude · Model: opus → claude-opus-5-5 · Effort: high · Cost: $0.79

  2. fullsend-ai-triage commented on Oct 24, 2026

    @fullsend-ai-triage

    Triage summary

    The request is fully specified and ready for implementation. No open PR in this repository addresses it, there are no duplicate issues, and nothing blocks it. The companion Fullsend research issue (fullsend#8789) is open, but it is parallel work, not a prerequisite.

    Facts verified during triage

    Implementation notes

    • Existing checks cover the scoped checks. The pre-commit hooks already handle this: experiment number, frontmatter and index linting, plus ruff, ty, bandit, private-key detection and a 1000 KB file-size limit. Run them only on the staged experiment files and README.md, as AGENTS.md says. No CI workflow change is needed; none of the current workflows run experiment tests.
    • Shared module language: the shared module is Go, while this repo's tooling is mostly Python. The issue requires reusing the pinned shared CLI/library rather than writing a parallel engine. Expect the setup step to fetch or build the pinned module, with scenario execution fully offline after that.
    • Artifact access: downloading Actions artifacts may need authentication and the artifacts expire. Build the record from public PR/run metadata and synthetic fixtures; don't depend on fetching artifacts.

    Proposed test case

    Fixed evaluation time, network disabled after setup:
    1. Clean candidate + synthetic 238/254 at 9900 bp  -> quality Failed, composed hold
    2. Same inputs at 9000 bp                          -> quality Passed, eligible, writer mock NOT called
    3. Exactly at threshold passes; one sample fewer fails (integer basis points)
    4. Synthetic GitLab consumer, approval + green CI, no risk assessment -> hold
    5. Required checks green + optional observed check failed -> OSPS-QA-03.01 hold;
       all relevant checks successful -> pass; pending/absent/incomplete data -> not pass
    6. Missing / stale / under-sampled / malformed / zero-population judge data -> Unknown/hold
    7. Candidate head mismatch or untrusted evidence -> hold
    8. Shadow mode or absent/expired authority with all assessments passing -> no writer action
    9. Candidate and merge-group subjects remain distinct
    10. Two runs produce identical output
    

    This issue is mainly a publication task: the reviewer must check the allow-listed historical extracts and scan the new files for internal identifiers before merge.


    Labels: This issue proposes a new experiment that is mostly documentation plus a small offline reproduction, matching how earlier experiment-creation issues were labeled.


    Next steps:

    • /fs-code — agent creates a PR to implement this issue
    • /fs-code <your instruction> — agent implements with your specific guidance
  3. ascerra commented on Oct 24, 2026

    @ascerra
    ContributorAuthor

    /fs-code

  4. fullsend-ai-coder commented on Oct 24, 2026

    @fullsend-ai-coder
    Contributor

    🤖 Finished Code · ✅ Success · Started 5:35 PM UTC · Completed 5:51 PM UTC

    Commit: 151b860 · View workflow run →

    Runtime: claude · Model: opus → claude-opus-5-5 · Effort: high · Cost: $4.24

  5. added 2 commits that reference this issue on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions