Bind TaskFlow stub-task call arguments in the Go SDK runtime - #70209
Merged
jason810496 merged 3 commits intoAug 20, 2026
Conversation
1 task
jason810496
force-pushed
the
feature/go-sdk/taskflow-arg-binding
branch
5 times, most recently
from
July 26, 2026 07:52
3f276fe to
25e2d12
Compare
1 task
1 task
jason810496
force-pushed
the
feature/go-sdk/taskflow-arg-binding
branch
5 times, most recently
from
August 18, 2026 02:33
9d95d0d to
3a2cad0
Compare
A Go task could only reach an upstream task's output by hand-writing a GetXCom call against a hard-coded task id, duplicating wiring the Dag file already owns and breaking silently whenever that upstream was renamed. apache#69757 ships the Python half: a `@task.stub` TaskFlow call is captured at Dag serialization as an ordered arg-binding spec and returned by ti_run. Consuming it here lets a Go task function take the Dag's literals and upstream XComs as ordinary typed parameters. A function declares either flat positional parameters or a single struct whose fields bind by name -- kwarg-style, so an unmatched field keeps its zero value while an argument no field claims fails the task. Signature problems are caught once at registration; per-execution arity, type and spec errors fail the task before its body runs, replacing a silent reflect.Zero fill.
Several binding problems stayed quiet until they were expensive or confusing. A parameter nesting an undecodable value failed on every execution rather than once when the bundle was built. A struct carrying `arg:` tags whose single argument no tag matched was decoded whole into the struct, so a typo'd tag surfaced as a decode error naming the Go type instead of the argument that matched nothing. A value_schema or from_default of the wrong wire shape was indistinguishable from an absent one, disabling the declared-type check or turning a captured stub default into an argument the author supposedly passed. The XCom whole-value decode and the concurrent multi-pull failure path were also reachable from a Dag but exercised only in their simplest shape, and the package docs re-explained the whole binding model at three sites, burying the rules they were meant to state.
jason810496
force-pushed
the
feature/go-sdk/taskflow-arg-binding
branch
2 times, most recently
from
August 18, 2026 08:51
1ddfc95 to
0507c8a
Compare
1 task
The Edge Worker's execution API carries no argument spec at all, so failing a task there for an argument-count mismatch blamed the Dag author for a limit of the transport. Keeping data parameters at their Go zero values is how those tasks behaved before binding existed, and that path is in maintenance rather than gaining the spec. Registration rejected struct shapes that decode without complaint -- a struct carrying a callback alongside its data never needed the callback filled -- and because registering a task panics, one such signature took its whole bundle down at startup rather than the single task. Adding a defaulted parameter to a stub is backwards compatible in Python, and has to stay so for the Go functions already bound to that stub: the captured default reaches the wire but needs no Go parameter to receive it. Untagged fields matched a Go field name verbatim, which no idiomatic snake_case stub parameter can produce, so tags were mandatory in practice and a mismatch quietly fell back to decoding the argument whole. Folding case and underscores makes the untagged form usable, and embedded structs now contribute their fields the way encoding/json has all along. A type that decodes itself from JSON also passed registration only to be rejected at run time by a schema check judging it on its Go kind.
jason810496
force-pushed
the
feature/go-sdk/taskflow-arg-binding
branch
from
August 19, 2026 06:08
0507c8a to
be8f718
Compare
jason810496
marked this pull request as ready for review
August 19, 2026 07:15
jason810496
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jscheffl and
potiuk
as code owners
August 19, 2026 07:15
uranusjr
reviewed
Aug 19, 2026
uranusjr
approved these changes
Aug 19, 2026
91 tasks
jason810496
commented
Aug 20, 2026
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Hi @ashb , I will merge this PR first to unblock the further works. Please feel free to raise any comment afterward and I will follow-up for them.
1 task
jason810496
added a commit
to jason810496/airflow
that referenced
this pull request
Aug 25, 2026
Add ADRs for the Mixed Lang Dag interface already shipped via apache#70209, the proposed Native Dag interface (apache#67155/apache#70158), and the common task constructs a native Go author will need next (TaskGroup, ShortCircuit, Branch, TriggerDagRun). Recording the rationale here gives reviewers and future contributors a single reference for these tradeoffs instead of reconstructing them from scattered PR discussions.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#69757 ships the Python-side contract: a
@task.stubTaskFlow call is serialized as an ordered arg-binding spec and returned byti_runasTIRunContext.arg_bindings. This PR makes the Go SDK actually consume it -- Go task functions receive the Dag file's literals and upstream XComs as typed parameters instead of hand-writingGetXComcalls with hard-coded upstream task ids:Supported TaskFlow syntax
The example bundle's
taskflow_binding_dag(go-sdk/dags/go_examples.py+go-sdk/example/bundle/taskflowbinding/) exercises the full surface end to end:On the Go side a task declares either flat positional data parameters or a single struct whose fields bind by name (mixing the two shapes is rejected at registration as too ambiguous):
Binding semantics, mirroring positional vs keyword calls:
fill every data parameter, with
Nonedecoded into nil-capable types.arg:tag. Unmatched fields keep their zero value,from_defaultentries may go unclaimed, and explicitly passed unclaimed arguments fail.not use this fallback.
schema mismatches, and strict struct decode errors are reported instead of silently zero-filling.
How
New
pkg/bindingpackage:Analyzeclassifies parameters once at registration (injectables vs JSON-decodable data parameters, or the single name-bound struct), recursing through nested types so an undecodable one fails when the bundle is built rather than on every execution.Resolvebinds the wire spec onto them. The wire union surfaces as a sealed sum type (binding.XComArg/binding.LiteralArg) whose variants andDataTypevocabulary are defined in terms of the generatedgenmodelsschema types, so the runtime types cannot drift from the wire model.Rejected Alternative
xcom:/xcom-key:struct tags were considered and dropped. A task that needs an extra XCom still asks the injected client explicitly.Was generative AI tooling used to co-author this PR?