Repository navigation
TensorFlow send/recv operations scheduling #43046
Description
Link to the documentation:
https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf
link to the OpKernels implementation:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/sendrecv_ops.h
Description of issue:
When a Tensorflow execute a graph, it partitions it into subgraphs s.t. each subgraph consists of nodes assigned to a single device. Value copy operations between those graphs are introduced in the form of Send and Recv nodes.
Send/Recv nodes are OpKernels that being executed on the CPU but trigger a memory copy operation (host to device or device to host). The framework support executing these operations in any order by using the Rendezvous object. Due to Recv1 OpKernel, we can be sure that OpKernel2 will be executed only when its input is ready. The question is about Send1 OpKernel. How does the framework schedule it? Since Send1 executed on CPU it should be executed immediately after calling to OpKernel1->compute in this case data dependency may be broken. Or maybe Send1 somehow belongs to the same subgraph as OpKernel1 even though they are being executed on different devices?
Activity
bhack commentedon Sep 8, 2020
/cc @keveman
rohan100jain commentedon Sep 9, 2020
This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there
Having said that - the diagram is inaccurate. Corresponding sends and recvs are on different devices - Send1 should be on Device1 and Recv2 should be on Device2. Send1 therefore has a data dependency on Op1 and it runs after that.
davidh600 commentedon Sep 9, 2020
@rohan100jain I have asked in StackOverflow too:
https://stackoverflow.com/questions/63793814/tensorflow-sour-code-send-recv-operations-scheduling
Doesn't Send/Recv ops CPU nodes? As far as know, for example, GPU doesn't have an implementation of this kernel.
Regarding the data dependency, there is a data dependency between OpKernel1 and OpKernel2 and still, we have to synchronize them with send and recv nodes.
PR #43046: [xla:gpu] Support simple scalar expressions in DynamicSlic…
PR #43046: [xla:gpu] Support simple scalar expressions in DynamicSlic…
PR #43046: [xla:gpu] Support simple scalar expressions in DynamicSlic…
PR #43046: [xla:gpu] Support simple scalar expressions in DynamicSlic…
PR #43046: [xla:gpu] Support simple scalar expressions in DynamicSlic…