From d4e54007a0fee68aa906f8bd578e01593a905fe0 Mon Sep 17 00:00:00 2001 From: Mark Shields Date: Mon, 20 Sep 2021 15:04:24 -0700 Subject: [PATCH 1/6] [checkpoint] first stab at summary --- .gitignore | 1 + 00xx-unified-target-and-device-planning.md | 198 +++++++++++++++++++++ 2 files changed, 199 insertions(+) create mode 100644 .gitignore create mode 100644 00xx-unified-target-and-device-planning.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..62c89355 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.idea/ \ No newline at end of file diff --git a/00xx-unified-target-and-device-planning.md b/00xx-unified-target-and-device-planning.md new file mode 100644 index 00000000..04ef2b1d --- /dev/null +++ b/00xx-unified-target-and-device-planning.md @@ -0,0 +1,198 @@ +- Feature Name: unified-target-device-planning +- Start Date: 2021-09-20 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +TVM currently has separate `Device` and a `Target` abstractions: + +- `Device` (aka `DLDevice`, provided by `dlpack`) is intended to be a runtime + abstraction describing a tensor's storage location. (It used to be called a + 'context', and some references to that term remain in the codebase.) It is + simply a pair of a `DLDeviceType` enum (eg `kDLCPU`, `kDLCUDA`, etc) and an + opaque integer representing a 'device identifier'. For the most part TVM + ignores or defaults the device identifier to zero so only the device type is + significant. Traditionally TVM used a single 'default' device for all calls + to (fused) primitive operators. + +- `Target` is intended to be a compile-time abstraction holding all the compiler + options which influence code-generation. For example `llvm + -mcpu=skylaxe-avx512`. Traditionally TVM only needed one target to be + specified so as to guide the compilation of all primitive operators. This is + because traditionally all tensor dataflow between primitive operators was + handled by some form of interpreter directly compiled into the runtime (the + 'graph executor', the 'interpreter', or the 'VM'), and traditionally only one + 'default' device was supported for the whole model. + +Two recent generalizations to TVM complicate this traditional view: +- TVM now supports 'hetrogenous' execution, in which each primitive operator + call may be executed and stored on a different device. The desired device + is indicated by an `on_device` 'annotation', which is just a call to a + built-in operator with an additional `device_type` attribute. A + 'device planning' analysis/pass associates a device with every primitive + operator call consistent with those annotations. Since different devices may + have very different compilation options, we also need a way to recover the + appropriate `Target` to handle each call. This is currently handled by: + - Building a `TargetMap` from `DLDeviceType` to `Target`, based on a list + of provided targets passed into the build API(s). + - Consulting that table for each call in `LowerTEPass`, using the + `DLDeviceType` associated to the call by device planning. +- TVM also now supports Ahead-of-time (AOT) compilation for the entire model. In + this world we can no longer assume everything other than calls to primitive + operators will be handled by the 'host' interpreter, and we must instead + clearly distinguish the `Target` for each primitive call from the `Target`(s) + handling the residual dataflow. This is currently handled by passing both a + 'host target' and traditional 'target' as a pair throughout the compiler, and + using a 'target annotation' pass to discover which code fragments should be + compiled for which targets. + +This has left us with a few problems: +1. We have independent 'device planning' and 'target annotation' passes, even + though: + - they are very similar (take some annotations or labels, find a + consistent assignment for all sub-expressions w.r.t. those annotations, + then find the transitions between devices/targets in the AST). + - they must be coherent, yet currently are handled completely independently + and probably can't be safetly used together. +2. We require all targets to have a unique `DLDeviceType`. However modern + platforms can have multiple 'CPU'-like devices (eg Arm 'big.LITTLE') and + multiple 'GPU'-like devices (eg an actual GPU and an Arm EthosU tensor + accelerator). Using device type alone to bridge the device and target worlds + is too rigid. +3. We have multiple conventions for representing available targets in the + codebase: + - a list of targets indexed by device type, using the invalid zero + device type to designate the 'default'. + - a 'host' and 'device' target pair, with calls to ensure they are + consistent with each other. + - a target containing a 'host target' field. + This makes the code particulaly difficult to maintain. + +It seems likely we'll need to support a first-class notion of 'memory scope' +as a refinement of 'device', but we don't want to do that until the above +issues are under control. + +In this RFC we propose: +1. We bring `Device` and `Target` together into a new structure: + ``` + class TargetDevice { + public: + Target target; + Device device; + } + ``` + (Eventually this would be extended to include a memory scope.) +2. We introduce a `TargetDevice` registry which maps globally unique + 'target device labels' to 'TargetDevice' objects. The registry may be + extended 'at compile time' by contrib code, and at runtime via `tvmc` + command line options. Conventions would be supported so that, e.g. + `my_accelerator:0` and `my_accelerator:1` (same architecture, but device ids + 0 and 1 respectively) could be referenced from "on_device" calls without + having to construct and register for the same `Target` twice. +3. We change "on_device" to use target device labels instead of device types. +4. We allow primitive operators to include `TargetDevice` annotations, for + example to specify they are available on only specific devices/targets. +5. We unify 'device planning' and 'target annotation' into a single + 'TargetDevice planning' pass. This pass understands: + - "on_device" + - Annotations on primitive ops + - The special handling for other built-ins, such as shape functions. +6. We remove all uses of target maps. For example, in `LowerTEPass` we + recover the target device label for each primitive operator call and use + the global label-to-TargetDevice map to recover the Target necessary to + complete compilation of the primitive. + +# Motivation +[motivation]: #motivation + +Why are we doing this? What use cases does it support? What is the expected outcome? + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Explain the proposal as if it was already included in the language and you were teaching it to a TVM user. + +That generally means: + +- Introducing new named concepts. +- Explaining what the feature enables (hint: think in terms of examples). +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. + +For internal RFCs (e.g. for compiler internals), this section should focus on how core contributors s +hould think about the change, and give examples of its concrete impact. + +For policy RFCs, this section should provide an example-driven introduction to the policy, + and explain its impact in concrete terms. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, +and explain more fully how the detailed proposal makes those examples work. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? + +# Prior art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- Does this feature exist in other ML compilers or languages and discuss the experince their community has had? +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? + If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +If there is no prior art, that is fine - your ideas are interesting to us whether they are + brand new or if it is an adaptation from other languages. + +Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. +Please also take into consideration that TVM intentionally diverges from other compilers. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future + independently of the solution that comes out of this RFC? + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect the language and project as a whole in a holistic +way. Try to use this section as a tool to more fully consider all possible +interactions with the project and language in your proposal. +Also consider how this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future RFC; such notes should be +in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. From 861366cf3e9e8636124c52fd495b0563871dc996 Mon Sep 17 00:00:00 2001 From: Mark Shields Date: Mon, 20 Sep 2021 15:05:00 -0700 Subject: [PATCH 2/6] [checkpoint] wrong dir --- .../00xx-unified-target-and-device-planning.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename 00xx-unified-target-and-device-planning.md => rfcs/00xx-unified-target-and-device-planning.md (100%) diff --git a/00xx-unified-target-and-device-planning.md b/rfcs/00xx-unified-target-and-device-planning.md similarity index 100% rename from 00xx-unified-target-and-device-planning.md rename to rfcs/00xx-unified-target-and-device-planning.md From dc0f50e8857d2ba530daad840616d4dd55b621e1 Mon Sep 17 00:00:00 2001 From: Mark Shields Date: Wed, 22 Sep 2021 15:43:31 -0700 Subject: [PATCH 3/6] rewrite with focus on CompileOptions instead of TargetDevice --- rfcs/00xx-improved-multi-target-handling.md | 184 ++++++++++++++++ ...00xx-unified-target-and-device-planning.md | 198 ------------------ 2 files changed, 184 insertions(+), 198 deletions(-) create mode 100644 rfcs/00xx-improved-multi-target-handling.md delete mode 100644 rfcs/00xx-unified-target-and-device-planning.md diff --git a/rfcs/00xx-improved-multi-target-handling.md b/rfcs/00xx-improved-multi-target-handling.md new file mode 100644 index 00000000..eaeb2987 --- /dev/null +++ b/rfcs/00xx-improved-multi-target-handling.md @@ -0,0 +1,184 @@ +- Feature Name: improved-multi-target-handling +- Start Date: 2021-09-20 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +Traditionally TVM managed two execution environments: +1. A device, such as a GPU, which would execute the 'inner' parts of fused primitive tensor operators. +2. A host, such as a CPU, which would a) execute any residual 'outer' parts of primitive tensor operators, and + b) coordinate the control and data flow between those operators. The later is managed by an executor (graph, + interpreter or VM) compiled directly into the TVM runtime. + +The compilation options for host and device are grouped into a pair of `Target` objects, typically called `target` and +`target_host`, which hold all the compiler flags and settings needed to influence code-generation. For example +`cuda` or `llvm -mcpu=skylaxe-avx512`. Device and host may be the same target. + +At runtime, tensors are managed by the `dlpack` library which provides a `Device` abstraction. That is a pair +of a `DLDeviceType` enum (eg `kDLCPU`, `kDLCUDA`, etc) and an opaque integer representing a 'device +identifier'. (However since TVM mostly ignores or defaults the device identifier to zero in effect TVM uses +`DLDeviceType` alone to identify devices.) Thus at runtime we also need a pair of `Device` objects +corresponding to our two `Target` objects. + +(Note that the codebase still refers to 'device' as 'context' in a few places.) + +Two more recent generalizations to TVM complicate this story: + +- TVM now supports 'hetrogenous' execution, in which each primitive operator call may be executed and + stored on a different device. The desired device is indicated by an `on_device` 'annotation', which is just + a call to a built-in operator with an additional `device_type` attribute. A 'device planning' analysis/pass + associates a device with every primitive operator call consistent with those annotations. + + Each call to a primitive operator for a particular `Device` signals we need to compile ('lower') that + primitive for the device, which requires a matching `Target`. This is currently handled by: + - Building a `TargetMap` from `DLDeviceType` to `Target`, based on a list of provided targets passed into + the build API(s). + - Consulting that table for each call in `LowerTEPass`, using the `DLDeviceType` associated to the call by + device planning. + - Using an entry for the invalid 'zero' device type to indicate the 'default' device. + + However TVM is being targetted to architectures with multiple CPUs (eg Arm 'Big.LITTLE') and + multiple devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). So we can no longer + assume a `DLDeviceType` uniquely identifies a device and it's appropriate `Target`. The `TargetMap` + convention also interacts poorly with the `target` and `target_host` convention, and the codebase + has gotten messy at those points. + +- TVM also now supports Ahead-of-time (AOT) compilation for the entire model rather than just the primitive + operators. This means we need to be explicit about the `Target` responsible for executing every Relay + sub-expression. Generally this is assumed to be the host target, however support for AOT has also required + support for Bring-your-own-compiler (BYOC) for 'embedded' targets. This has resulted in a compilation + flow very similar to device planning whereby `Target` annotations are used to decide which 'compiler' is + to be used for each Relay sub-expression. + + However this machinely works independently of the above device planning, and it's not clear how they + would ever interact. The gap between `Target` and `Device` make reconciliation difficult. + +In this RFC we propose: +1. Use a combination of `Target` and `Device` as the unit of planning in 'device planning': + ``` + class TargetDevice { + public: + Target target; + Device device; + } + ``` + (Eventually this would be extended to include a memory scope.) +2. Allow `TargetDevice` objects to be registered under a globally unique `TargetDeviceLabel` (ie a + string). Registration may be 'static' (ie built into the TVM compiler) or 'dynamic' (ie injected for a + particular run of the compiler, eg on the `tvmc` command line). +3. We change the "on_device" and "device_copy" call attributes to use `TargetDeviceLabel`s instead + of integers (ie device types). +4. We allow primitive operators to include (sets of) `TargetDeviceLabel`s, for example to specify they are + available only on specific devices/targets. +5. We remove all uses of target maps. For example, in `LowerTEPass` we + recover the `TargetDeviceLabel` for each primitive operator call and use the global `TargetDevice` registry + to recover the `Target` necessary to complete compilation of the primitive, and the `Device`s needed to effect + any tensor copies. +6. We gather the various `Target` and `Device` defaults into a single `CompileOptions` class: + - The default `TargetDeviceLabel` for primitive operators. + - The default `TargetDeviceLabel` for non primitive operators, such as + Relay control flow and shape computation. +7. We remove the various copies of target/target_host reconciliation, TargetMap + construction and 'default/fallback' device calculation from the codebase in favor + of the centralized `CompileOptions` class. +8. We attach the `CompileOptions` class to an `IRModule` attribute. + +We stop short of actually changing the current BYOC TargetAnnotation machinery. But our intent is to +at least remove as many accidental differences to make the next step clear. + +-------- rest still in template form -------- + +# Motivation +[motivation]: #motivation + +Why are we doing this? What use cases does it support? What is the expected outcome? + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Explain the proposal as if it was already included in the language and you were teaching it to a TVM user. + +That generally means: + +- Introducing new named concepts. +- Explaining what the feature enables (hint: think in terms of examples). +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. + +For internal RFCs (e.g. for compiler internals), this section should focus on how core contributors s +hould think about the change, and give examples of its concrete impact. + +For policy RFCs, this section should provide an example-driven introduction to the policy, + and explain its impact in concrete terms. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, +and explain more fully how the detailed proposal makes those examples work. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? + +# Prior art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- Does this feature exist in other ML compilers or languages and discuss the experince their community has had? +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? + If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +If there is no prior art, that is fine - your ideas are interesting to us whether they are + brand new or if it is an adaptation from other languages. + +Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. +Please also take into consideration that TVM intentionally diverges from other compilers. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future + independently of the solution that comes out of this RFC? + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect the language and project as a whole in a holistic +way. Try to use this section as a tool to more fully consider all possible +interactions with the project and language in your proposal. +Also consider how this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future RFC; such notes should be +in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. diff --git a/rfcs/00xx-unified-target-and-device-planning.md b/rfcs/00xx-unified-target-and-device-planning.md deleted file mode 100644 index 04ef2b1d..00000000 --- a/rfcs/00xx-unified-target-and-device-planning.md +++ /dev/null @@ -1,198 +0,0 @@ -- Feature Name: unified-target-device-planning -- Start Date: 2021-09-20 -- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) -- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) - -# Summary -[summary]: #summary - -TVM currently has separate `Device` and a `Target` abstractions: - -- `Device` (aka `DLDevice`, provided by `dlpack`) is intended to be a runtime - abstraction describing a tensor's storage location. (It used to be called a - 'context', and some references to that term remain in the codebase.) It is - simply a pair of a `DLDeviceType` enum (eg `kDLCPU`, `kDLCUDA`, etc) and an - opaque integer representing a 'device identifier'. For the most part TVM - ignores or defaults the device identifier to zero so only the device type is - significant. Traditionally TVM used a single 'default' device for all calls - to (fused) primitive operators. - -- `Target` is intended to be a compile-time abstraction holding all the compiler - options which influence code-generation. For example `llvm - -mcpu=skylaxe-avx512`. Traditionally TVM only needed one target to be - specified so as to guide the compilation of all primitive operators. This is - because traditionally all tensor dataflow between primitive operators was - handled by some form of interpreter directly compiled into the runtime (the - 'graph executor', the 'interpreter', or the 'VM'), and traditionally only one - 'default' device was supported for the whole model. - -Two recent generalizations to TVM complicate this traditional view: -- TVM now supports 'hetrogenous' execution, in which each primitive operator - call may be executed and stored on a different device. The desired device - is indicated by an `on_device` 'annotation', which is just a call to a - built-in operator with an additional `device_type` attribute. A - 'device planning' analysis/pass associates a device with every primitive - operator call consistent with those annotations. Since different devices may - have very different compilation options, we also need a way to recover the - appropriate `Target` to handle each call. This is currently handled by: - - Building a `TargetMap` from `DLDeviceType` to `Target`, based on a list - of provided targets passed into the build API(s). - - Consulting that table for each call in `LowerTEPass`, using the - `DLDeviceType` associated to the call by device planning. -- TVM also now supports Ahead-of-time (AOT) compilation for the entire model. In - this world we can no longer assume everything other than calls to primitive - operators will be handled by the 'host' interpreter, and we must instead - clearly distinguish the `Target` for each primitive call from the `Target`(s) - handling the residual dataflow. This is currently handled by passing both a - 'host target' and traditional 'target' as a pair throughout the compiler, and - using a 'target annotation' pass to discover which code fragments should be - compiled for which targets. - -This has left us with a few problems: -1. We have independent 'device planning' and 'target annotation' passes, even - though: - - they are very similar (take some annotations or labels, find a - consistent assignment for all sub-expressions w.r.t. those annotations, - then find the transitions between devices/targets in the AST). - - they must be coherent, yet currently are handled completely independently - and probably can't be safetly used together. -2. We require all targets to have a unique `DLDeviceType`. However modern - platforms can have multiple 'CPU'-like devices (eg Arm 'big.LITTLE') and - multiple 'GPU'-like devices (eg an actual GPU and an Arm EthosU tensor - accelerator). Using device type alone to bridge the device and target worlds - is too rigid. -3. We have multiple conventions for representing available targets in the - codebase: - - a list of targets indexed by device type, using the invalid zero - device type to designate the 'default'. - - a 'host' and 'device' target pair, with calls to ensure they are - consistent with each other. - - a target containing a 'host target' field. - This makes the code particulaly difficult to maintain. - -It seems likely we'll need to support a first-class notion of 'memory scope' -as a refinement of 'device', but we don't want to do that until the above -issues are under control. - -In this RFC we propose: -1. We bring `Device` and `Target` together into a new structure: - ``` - class TargetDevice { - public: - Target target; - Device device; - } - ``` - (Eventually this would be extended to include a memory scope.) -2. We introduce a `TargetDevice` registry which maps globally unique - 'target device labels' to 'TargetDevice' objects. The registry may be - extended 'at compile time' by contrib code, and at runtime via `tvmc` - command line options. Conventions would be supported so that, e.g. - `my_accelerator:0` and `my_accelerator:1` (same architecture, but device ids - 0 and 1 respectively) could be referenced from "on_device" calls without - having to construct and register for the same `Target` twice. -3. We change "on_device" to use target device labels instead of device types. -4. We allow primitive operators to include `TargetDevice` annotations, for - example to specify they are available on only specific devices/targets. -5. We unify 'device planning' and 'target annotation' into a single - 'TargetDevice planning' pass. This pass understands: - - "on_device" - - Annotations on primitive ops - - The special handling for other built-ins, such as shape functions. -6. We remove all uses of target maps. For example, in `LowerTEPass` we - recover the target device label for each primitive operator call and use - the global label-to-TargetDevice map to recover the Target necessary to - complete compilation of the primitive. - -# Motivation -[motivation]: #motivation - -Why are we doing this? What use cases does it support? What is the expected outcome? - -# Guide-level explanation -[guide-level-explanation]: #guide-level-explanation - -Explain the proposal as if it was already included in the language and you were teaching it to a TVM user. - -That generally means: - -- Introducing new named concepts. -- Explaining what the feature enables (hint: think in terms of examples). -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. - -For internal RFCs (e.g. for compiler internals), this section should focus on how core contributors s -hould think about the change, and give examples of its concrete impact. - -For policy RFCs, this section should provide an example-driven introduction to the policy, - and explain its impact in concrete terms. - -# Reference-level explanation -[reference-level-explanation]: #reference-level-explanation - -This is the technical portion of the RFC. Explain the design in sufficient detail that: - -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. - -The section should return to the examples given in the previous section, -and explain more fully how the detailed proposal makes those examples work. - -# Drawbacks -[drawbacks]: #drawbacks - -Why should we *not* do this? - -# Rationale and alternatives -[rationale-and-alternatives]: #rationale-and-alternatives - -- Why is this design the best in the space of possible designs? -- What other designs have been considered and what is the rationale for not choosing them? -- What is the impact of not doing this? - -# Prior art -[prior-art]: #prior-art - -Discuss prior art, both the good and the bad, in relation to this proposal. -A few examples of what this can include are: - -- Does this feature exist in other ML compilers or languages and discuss the experince their community has had? -- For community proposals: Is this done by some other community and what were their experiences with it? -- For other teams: What lessons can we learn from what other communities have done here? -- Papers: Are there any published papers or great posts that discuss this? - If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. - -If there is no prior art, that is fine - your ideas are interesting to us whether they are - brand new or if it is an adaptation from other languages. - -Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. -Please also take into consideration that TVM intentionally diverges from other compilers. - -# Unresolved questions -[unresolved-questions]: #unresolved-questions - -- What parts of the design do you expect to resolve through the RFC process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this RFC that could be addressed in the future - independently of the solution that comes out of this RFC? - -# Future possibilities -[future-possibilities]: #future-possibilities - -Think about what the natural extension and evolution of your proposal would -be and how it would affect the language and project as a whole in a holistic -way. Try to use this section as a tool to more fully consider all possible -interactions with the project and language in your proposal. -Also consider how this all fits into the roadmap for the project -and of the relevant sub-team. - -This is also a good place to "dump ideas", if they are out of scope for the -RFC you are writing but otherwise related. - -If you have tried and cannot think of any future possibilities, -you may simply state that you cannot think of anything. - -Note that having something written down in the future-possibilities section -is not a reason to accept the current or a future RFC; such notes should be -in the section on motivation or rationale in this or subsequent RFCs. -The section merely provides additional information. From b3809e9b6615b788812d8f04975e42e4db0ca0b7 Mon Sep 17 00:00:00 2001 From: Mark Shields Date: Tue, 28 Sep 2021 18:13:32 -0700 Subject: [PATCH 4/6] Another rewrite. --- rfcs/00xx-improved-multi-target-handling.md | 141 +++++++++----------- 1 file changed, 66 insertions(+), 75 deletions(-) diff --git a/rfcs/00xx-improved-multi-target-handling.md b/rfcs/00xx-improved-multi-target-handling.md index eaeb2987..385beb49 100644 --- a/rfcs/00xx-improved-multi-target-handling.md +++ b/rfcs/00xx-improved-multi-target-handling.md @@ -6,87 +6,78 @@ # Summary [summary]: #summary -Traditionally TVM managed two execution environments: -1. A device, such as a GPU, which would execute the 'inner' parts of fused primitive tensor operators. -2. A host, such as a CPU, which would a) execute any residual 'outer' parts of primitive tensor operators, and - b) coordinate the control and data flow between those operators. The later is managed by an executor (graph, - interpreter or VM) compiled directly into the TVM runtime. - -The compilation options for host and device are grouped into a pair of `Target` objects, typically called `target` and -`target_host`, which hold all the compiler flags and settings needed to influence code-generation. For example -`cuda` or `llvm -mcpu=skylaxe-avx512`. Device and host may be the same target. - -At runtime, tensors are managed by the `dlpack` library which provides a `Device` abstraction. That is a pair -of a `DLDeviceType` enum (eg `kDLCPU`, `kDLCUDA`, etc) and an opaque integer representing a 'device -identifier'. (However since TVM mostly ignores or defaults the device identifier to zero in effect TVM uses -`DLDeviceType` alone to identify devices.) Thus at runtime we also need a pair of `Device` objects -corresponding to our two `Target` objects. - -(Note that the codebase still refers to 'device' as 'context' in a few places.) - -Two more recent generalizations to TVM complicate this story: - -- TVM now supports 'hetrogenous' execution, in which each primitive operator call may be executed and - stored on a different device. The desired device is indicated by an `on_device` 'annotation', which is just - a call to a built-in operator with an additional `device_type` attribute. A 'device planning' analysis/pass - associates a device with every primitive operator call consistent with those annotations. - - Each call to a primitive operator for a particular `Device` signals we need to compile ('lower') that - primitive for the device, which requires a matching `Target`. This is currently handled by: - - Building a `TargetMap` from `DLDeviceType` to `Target`, based on a list of provided targets passed into - the build API(s). - - Consulting that table for each call in `LowerTEPass`, using the `DLDeviceType` associated to the call by - device planning. - - Using an entry for the invalid 'zero' device type to indicate the 'default' device. - - However TVM is being targetted to architectures with multiple CPUs (eg Arm 'Big.LITTLE') and - multiple devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). So we can no longer - assume a `DLDeviceType` uniquely identifies a device and it's appropriate `Target`. The `TargetMap` - convention also interacts poorly with the `target` and `target_host` convention, and the codebase - has gotten messy at those points. - -- TVM also now supports Ahead-of-time (AOT) compilation for the entire model rather than just the primitive - operators. This means we need to be explicit about the `Target` responsible for executing every Relay - sub-expression. Generally this is assumed to be the host target, however support for AOT has also required - support for Bring-your-own-compiler (BYOC) for 'embedded' targets. This has resulted in a compilation - flow very similar to device planning whereby `Target` annotations are used to decide which 'compiler' is - to be used for each Relay sub-expression. - - However this machinely works independently of the above device planning, and it's not clear how they - would ever interact. The gap between `Target` and `Device` make reconciliation difficult. - -In this RFC we propose: -1. Use a combination of `Target` and `Device` as the unit of planning in 'device planning': +TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than +one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows: +1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should + reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc). +2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression, + including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default' + device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries. +3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap` + from `DLDeviceType` to `Target` for all of those objects. +4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that + primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`. + +This approach has 5 problems: +1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly + devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer + uniquely determines a `Target`. +2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an + arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators. + Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system. +3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for + primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the + overall Relay control-flow. +4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations + should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied + `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that + `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets, + it is particularly important to ensure there's a single source of truth for available `Target`s. +5. TVM also supports a 'BYOC' extension mechanism. This allows "target." annotations to be placed on + primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target + annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph + pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this + machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate + concept from `Target` objects. + +In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope +is for this RFC to clear the way to taking on 5 in the future. + +Our proposal is: +1. Extend `Target` to have a `DLDeviceType` attribute. +2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie + built into the TVM compiler via another REGISTER macro) and 'dynamic' (ie injected for a particular run of the + compiler, eg as part of `tvmc` command line processing). (This machinery should be reconciled with the existing + CUDA-specific target registration map.) +3. Change the "on_device" call attributes to use a string instead of an integers (ie `DLDeviceType`). The string + can be of the form `` or `:`. The former simply implies a device id of 0. +4. Rework device planning to use a pair of `Target` and 'device id' instead of `DLDeviceType`: ``` class TargetDevice { public: Target target; - Device device; + int device_id; } ``` - (Eventually this would be extended to include a memory scope.) -2. Allow `TargetDevice` objects to be registered under a globally unique `TargetDeviceLabel` (ie a - string). Registration may be 'static' (ie built into the TVM compiler) or 'dynamic' (ie injected for a - particular run of the compiler, eg on the `tvmc` command line). -3. We change the "on_device" and "device_copy" call attributes to use `TargetDeviceLabel`s instead - of integers (ie device types). -4. We allow primitive operators to include (sets of) `TargetDeviceLabel`s, for example to specify they are - available only on specific devices/targets. -5. We remove all uses of target maps. For example, in `LowerTEPass` we - recover the `TargetDeviceLabel` for each primitive operator call and use the global `TargetDevice` registry - to recover the `Target` necessary to complete compilation of the primitive, and the `Device`s needed to effect - any tensor copies. -6. We gather the various `Target` and `Device` defaults into a single `CompileOptions` class: - - The default `TargetDeviceLabel` for primitive operators. - - The default `TargetDeviceLabel` for non primitive operators, such as - Relay control flow and shape computation. -7. We remove the various copies of target/target_host reconciliation, TargetMap - construction and 'default/fallback' device calculation from the codebase in favor - of the centralized `CompileOptions` class. -8. We attach the `CompileOptions` class to an `IRModule` attribute. - -We stop short of actually changing the current BYOC TargetAnnotation machinery. But our intent is to -at least remove as many accidental differences to make the next step clear. + (We could also use a `Device` and accept the redundant `DLDeviceType` specification.) It is trivial + to go from an "on_device" label to a `TargetDevice` and back using the global `Target` registry. +5. Remove all uses of `TargetMap`. For example, in `LowerTEPass` we simply use the `TargetDevice` associated with + every primitive operator call already found by device planning. +6. Bind two `TargetDevice`s as attributes on every `IRModule`: + - The default for primitive operators not otherwise constrained by "on_device" annotations. + - The default for non primitive operators, such as Relay control flow and shape computation. +7. We remove the various copies of target/target_host reconciliation, `TargetMap` + construction and 'default/fallback' device calculation from the codebase. + +This proposal tackles the original problems: +1. There's now no ambiguity about `Targets` since we propagate them from the global registry directly. +2. We support device ids. +3. We always know the `Target` for every sub-expression and don't need to pass around the `target` and + `target host` separately. +4. `Targets` are never created on the fly, they are first registered then propagated. +5. The global registration implied by the existing BYOC target names is now more similar to how the mainline + `Target`s are handled. + -------- rest still in template form -------- From e338d52b7ff974324202a3359a50544ad56c1db9 Mon Sep 17 00:00:00 2001 From: Mark Shields Date: Tue, 28 Sep 2021 18:29:56 -0700 Subject: [PATCH 5/6] Formatting --- rfcs/00xx-improved-multi-target-handling.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/rfcs/00xx-improved-multi-target-handling.md b/rfcs/00xx-improved-multi-target-handling.md index 385beb49..0a8a8a2b 100644 --- a/rfcs/00xx-improved-multi-target-handling.md +++ b/rfcs/00xx-improved-multi-target-handling.md @@ -27,13 +27,14 @@ This approach has 5 problems: Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system. 3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the - overall Relay control-flow. + overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to + deal with these different conventions. 4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets, it is particularly important to ensure there's a single source of truth for available `Target`s. -5. TVM also supports a 'BYOC' extension mechanism. This allows "target." annotations to be placed on +5. TVM also supports a 'BYOC' extension mechanism. This allows `"target."` annotations to be placed on primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this From 19a40c617ec1fbd7f8eb4de08a3dad6264812b4a Mon Sep 17 00:00:00 2001 From: Mark Shields Date: Thu, 21 Oct 2021 12:09:52 -0700 Subject: [PATCH 6/6] rewrite to take on BYOC and downplay target management --- ...device-target-and-memory-scope-planning.md | 244 ++++++++++++++++++ rfcs/00xx-improved-multi-target-handling.md | 176 ------------- 2 files changed, 244 insertions(+), 176 deletions(-) create mode 100644 rfcs/0038-unified-device-target-and-memory-scope-planning.md delete mode 100644 rfcs/00xx-improved-multi-target-handling.md diff --git a/rfcs/0038-unified-device-target-and-memory-scope-planning.md b/rfcs/0038-unified-device-target-and-memory-scope-planning.md new file mode 100644 index 00000000..db76b716 --- /dev/null +++ b/rfcs/0038-unified-device-target-and-memory-scope-planning.md @@ -0,0 +1,244 @@ +- Feature Name: unified-target-device-and-memory-scope-planning +- Start Date: 2021-09-20 +- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038) +- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327) + +# Summary +[summary]: #summary + +TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated +on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows: +1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result + should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc). +2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay + sub-expression, including every primitive operator call. Sub-expressions which are unconstrained + are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data + needs to cross device boundaries. +3. The user must also supply a list of `Target` objects. The compiler uses that list to build + a `TargetMap` from `DLDeviceType` to `Target`. +4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile + ('lower') that primitive for that device. The `Target` to use for that compilation is found from + the `TargetMap` by the `LowerTEPass`. + +For the BYOC flow things are quite different: +1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular + `target.`. Here `` serves only to distinguish possible BYOC toolchain names and is + currently not connected to the `Target` machinery in any way. The function should return true if + the given expression could be compiled for toolchain ``. (However there are currently no + examples of this annotation in-tree.) +2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions + which have been hoisted out of a larger expression based on a fusion pattern. The attribute can + have any value of the form `"some.arbitrary.prefix."`. Again, this indicates the function + could be compiled for toolchain ``. (The EthosU compilation flow illustrates this approach + in-tree.) +3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique + toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in + to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end` + annotations. +4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end` + annotations into new top-level `Function`s with a `"Compiler"` attribute bound to the toolchain + ``. +5. The rest of the compilation flow treats `"Compiler"` annotated functions specially. + +We have 6 problems: +1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple + tensor-friendly devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a + `DLDeviceType` no longer uniquely determines a `Target`. +2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a + `DLDeviceType` and an arbitrary 'device id', TVM does not consistently plumb the device id + through annotations, passes and operators. Thus currently we cannot use 'device id' to + distinguish, eg, two CPUs in the same system. +3. Upcoming work requires us to distinguish and propagate memory scopes for data at the Relay + level. (See also [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md) + which has a similar need for memory scope propagation at the TIR level). This is an identical + problem to propagating devices, and it seems most natural to simply combine targets, devices and + memory scopes into a single 'target of device planing' rather than implementing a whole new pass. +4. Device planning currently has no machinery to hoist adjacent expressions which share the same device + into their own Relay `Function`. For all our executors except VM that's unnecessary anyway since + all Relay expressions left over after lowering are interpreted by the runtime. However for AOT we + have to compile *all* Relay code for a particular target. Note the BOYC machinery does support this, + but for the purposes of redirecting the compilation flow entirely. We need a middle ground. +5. The BYOC flow is not connected to the `Target` machinery in any way. +6. The BYOC annotate/partition flow is very similar to the device annotate/rewrite flow. For comparison: + + | Feature | Device Planning | BYOC | + | --------------------- | -------------------------- | ----------------------------------------------- | + | Source of annotations | `on_device`, `device_copy` | `FTVMAnnotateTarget`, `MergeComposite`+patterns | + | Target of planning | DLDeviceType | Toolchain name | + | Propagation | Unification based | Ad-hoc | + | Relay support | Full | First-order, no ADTs | + | Delimiting | insert `device_copy` | insert `compiler_begin`, `compiler_end` | + | Multiple per expr | No | Yes (though always picks first) | + | Hoists into functions | No | Yes | + | Customized heuristics | No | No | + + Taking the 'upper bound' of the two implementations seems ideal, especially to address issues 4 (limitation + of device planning) and 5 (limitation of BYOC) above. + +Our proposal is: +1. We introduce a new FFI-friendly class to represent a *S*torage or *E*xecution *Scope*: + + ``` + class SEScope { + DLDeviceType device_type; + int virtual_device_id; + Target target; + String memory_scope; + } + ``` + + We allow each of these fields to be independently 'constrained' (ie have a specific value) or + 'unconstrained' (no specific value for the field is known yet). In particular, it is valid for + an `SEScope` to contain only a `device_type`. However if the `target` field is defined then + `device_type` must equal `target->kind->device_type`. + +2. At this stage we leave the `memory_scope` field uninterpreted. For example, we don't attempt to + represent that, eg, `"global"` on a `kDLCPU` is the same memory area as `"host"` on a `kDLCUDA` and thus no + `device_copy` operation is required between those scopes. We'll pick this issue up again after + [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md) + has landed. + +3. The `on_device` and `device_copy` call attributes use `SEScope`s instead of integers. However the Python + bindings for these 'operators' continue to accept a `Device` for convenience. The machinery in `LowerTEPass` + which resolves `DLDeviceTypes` to `Targets` is moved up in the compilation flow and becomes part of + `PlanDevices`. In particular, any `SEScope` encountered during device planning is 'canonicalized' to fill + in a `Target` by the same lookup as we do today. This means we continue to support the easy shorthand of + referring to devices by the `DLDeviceType` alone. However, advanced users can supply a `SEScope` to these + operators which contains the exact `Target` to use. + +4. We rework device planning to be in terms of `SEScope`s instead of `DLDeviceTypes`. Two `SEScope`s + become special: + - We need a default scope for all primitive operators which are not otherwise + constrained to a particular scope. + - We need a scope for 'host-only' operations and data, such as for shapes and shape functions. + (Currently this is hardcoded to `kDLCPU`). + +5. We extend `PlanDevices` to be able to a) run *after* lowering and b) refine existing constraints. It will + look inside calls to `PrimFunc`s and follow the chain: + + ``` + tir::PrimFunc.buffer_map -> tir::Buffer.data -> tir::Var.type_annotation -> PointerType.storage_scope -> String + ``` + + to discover the memory scope for each Relay argument. That scope will enter `SEScope`s and flow through the + existing unification machinery. The existing sub-pass in `PlanDevices` will insert `device_copy` calls + wherever sub-expressions disagree on their memory scope. + + (An additional pass is planned to heuristically move `device_copy`s around, and eliminate redundant + copies, however that's outside the scope of this RFC.) + +6. We rework `PartitionGraph` to `PartitionBySEScope` to work on `SEScope` annotations instead of + `compiler_begin` and `compiler_end` annotations. Algorithmically it's not a big change -- maximal + sub-expressions which share the same `SEScope` (or a projection thereof, eg just the `target`) are hoisted + into global `Function`s. The function's `"result_se_scope"` attribute describes both the scope holding the + function's result *and* the `Target` for which the function is to be compiled. + +7. We allow `MergeComposite` to be used to insert `on_device` annotations, call it `MergeAndAnnotate`. + +8. (?) We rework `AnnotateTarget` to just look for `FTVMAnnotateTarget` operator attributes, call it + `AnnotateSEScopes`. When the function fires an `on_device` annotation is inserted. However since + there are no examples of these attributes being used in-tree perhaps this is dead code? + +9. (?) We rework `PlanDevices` to support collecting multiple candidate `SEScopes`, mimicking the + current behavior in `AnnotateTarget`. However, since the current behavior simply picks the + first toolchain name, and we don't currently have any passes which attempt to solve the + (very hard) device selection problem, this work may be best deferred till we understand more. + +10. We retire the BYOC `MergeComposite`/`AnnotateTarget`/`PartitionGraph` flow in favor of the + `MergeAndAnnotate`/`AnnotateSEScopes`/`PlanDevices`/`PartitionBySEScope` flow. BYOC hooks which + are currently keyed by toolchain name can instead be keyed by `Target`. + +-------- rest still in template form -------- + +# Motivation +[motivation]: #motivation + +Why are we doing this? What use cases does it support? What is the expected outcome? + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Explain the proposal as if it was already included in the language and you were teaching it to a TVM user. + +That generally means: + +- Introducing new named concepts. +- Explaining what the feature enables (hint: think in terms of examples). +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. + +For internal RFCs (e.g. for compiler internals), this section should focus on how core contributors s +hould think about the change, and give examples of its concrete impact. + +For policy RFCs, this section should provide an example-driven introduction to the policy, + and explain its impact in concrete terms. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, +and explain more fully how the detailed proposal makes those examples work. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? + +# Prior art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- Does this feature exist in other ML compilers or languages and discuss the experince their community has had? +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? + If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +If there is no prior art, that is fine - your ideas are interesting to us whether they are + brand new or if it is an adaptation from other languages. + +Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. +Please also take into consideration that TVM intentionally diverges from other compilers. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future + independently of the solution that comes out of this RFC? + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect the language and project as a whole in a holistic +way. Try to use this section as a tool to more fully consider all possible +interactions with the project and language in your proposal. +Also consider how this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future RFC; such notes should be +in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. diff --git a/rfcs/00xx-improved-multi-target-handling.md b/rfcs/00xx-improved-multi-target-handling.md deleted file mode 100644 index 0a8a8a2b..00000000 --- a/rfcs/00xx-improved-multi-target-handling.md +++ /dev/null @@ -1,176 +0,0 @@ -- Feature Name: improved-multi-target-handling -- Start Date: 2021-09-20 -- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) -- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) - -# Summary -[summary]: #summary - -TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than -one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows: -1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should - reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc). -2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression, - including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default' - device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries. -3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap` - from `DLDeviceType` to `Target` for all of those objects. -4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that - primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`. - -This approach has 5 problems: -1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly - devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer - uniquely determines a `Target`. -2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an - arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators. - Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system. -3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for - primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the - overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to - deal with these different conventions. -4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations - should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied - `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that - `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets, - it is particularly important to ensure there's a single source of truth for available `Target`s. -5. TVM also supports a 'BYOC' extension mechanism. This allows `"target."` annotations to be placed on - primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target - annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph - pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this - machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate - concept from `Target` objects. - -In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope -is for this RFC to clear the way to taking on 5 in the future. - -Our proposal is: -1. Extend `Target` to have a `DLDeviceType` attribute. -2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie - built into the TVM compiler via another REGISTER macro) and 'dynamic' (ie injected for a particular run of the - compiler, eg as part of `tvmc` command line processing). (This machinery should be reconciled with the existing - CUDA-specific target registration map.) -3. Change the "on_device" call attributes to use a string instead of an integers (ie `DLDeviceType`). The string - can be of the form `` or `:`. The former simply implies a device id of 0. -4. Rework device planning to use a pair of `Target` and 'device id' instead of `DLDeviceType`: - ``` - class TargetDevice { - public: - Target target; - int device_id; - } - ``` - (We could also use a `Device` and accept the redundant `DLDeviceType` specification.) It is trivial - to go from an "on_device" label to a `TargetDevice` and back using the global `Target` registry. -5. Remove all uses of `TargetMap`. For example, in `LowerTEPass` we simply use the `TargetDevice` associated with - every primitive operator call already found by device planning. -6. Bind two `TargetDevice`s as attributes on every `IRModule`: - - The default for primitive operators not otherwise constrained by "on_device" annotations. - - The default for non primitive operators, such as Relay control flow and shape computation. -7. We remove the various copies of target/target_host reconciliation, `TargetMap` - construction and 'default/fallback' device calculation from the codebase. - -This proposal tackles the original problems: -1. There's now no ambiguity about `Targets` since we propagate them from the global registry directly. -2. We support device ids. -3. We always know the `Target` for every sub-expression and don't need to pass around the `target` and - `target host` separately. -4. `Targets` are never created on the fly, they are first registered then propagated. -5. The global registration implied by the existing BYOC target names is now more similar to how the mainline - `Target`s are handled. - - --------- rest still in template form -------- - -# Motivation -[motivation]: #motivation - -Why are we doing this? What use cases does it support? What is the expected outcome? - -# Guide-level explanation -[guide-level-explanation]: #guide-level-explanation - -Explain the proposal as if it was already included in the language and you were teaching it to a TVM user. - -That generally means: - -- Introducing new named concepts. -- Explaining what the feature enables (hint: think in terms of examples). -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. - -For internal RFCs (e.g. for compiler internals), this section should focus on how core contributors s -hould think about the change, and give examples of its concrete impact. - -For policy RFCs, this section should provide an example-driven introduction to the policy, - and explain its impact in concrete terms. - -# Reference-level explanation -[reference-level-explanation]: #reference-level-explanation - -This is the technical portion of the RFC. Explain the design in sufficient detail that: - -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. - -The section should return to the examples given in the previous section, -and explain more fully how the detailed proposal makes those examples work. - -# Drawbacks -[drawbacks]: #drawbacks - -Why should we *not* do this? - -# Rationale and alternatives -[rationale-and-alternatives]: #rationale-and-alternatives - -- Why is this design the best in the space of possible designs? -- What other designs have been considered and what is the rationale for not choosing them? -- What is the impact of not doing this? - -# Prior art -[prior-art]: #prior-art - -Discuss prior art, both the good and the bad, in relation to this proposal. -A few examples of what this can include are: - -- Does this feature exist in other ML compilers or languages and discuss the experince their community has had? -- For community proposals: Is this done by some other community and what were their experiences with it? -- For other teams: What lessons can we learn from what other communities have done here? -- Papers: Are there any published papers or great posts that discuss this? - If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. - -If there is no prior art, that is fine - your ideas are interesting to us whether they are - brand new or if it is an adaptation from other languages. - -Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. -Please also take into consideration that TVM intentionally diverges from other compilers. - -# Unresolved questions -[unresolved-questions]: #unresolved-questions - -- What parts of the design do you expect to resolve through the RFC process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this RFC that could be addressed in the future - independently of the solution that comes out of this RFC? - -# Future possibilities -[future-possibilities]: #future-possibilities - -Think about what the natural extension and evolution of your proposal would -be and how it would affect the language and project as a whole in a holistic -way. Try to use this section as a tool to more fully consider all possible -interactions with the project and language in your proposal. -Also consider how this all fits into the roadmap for the project -and of the relevant sub-team. - -This is also a good place to "dump ideas", if they are out of scope for the -RFC you are writing but otherwise related. - -If you have tried and cannot think of any future possibilities, -you may simply state that you cannot think of anything. - -Note that having something written down in the future-possibilities section -is not a reason to accept the current or a future RFC; such notes should be -in the section on motivation or rationale in this or subsequent RFCs. -The section merely provides additional information.