You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Question about multi streams operate the same QP and CQ #48
If I use multiple streams in different threads to operate the same SQ or SCQ, how to make sure to get the right DB or CQ values ? Thanks for your answer
This is a very complex topic. The descriptors must be submitted to the CUDA streams in such a way that they will be triggered in the expected order, I.e. the same one in which they have been gotten back from the APIs.
Thank you for your answer. I wonder if using spin lock before exp post for SQ and unlock after getting the descriptors for SCQ can resolve this problem about accessing the same SQ and SCQ ?
I am not sure...
note that libgdsync APIs are not thread safe, due to the use of custom res_domain.
CPU locks would only protect the calls to IB verbs API and would result in a certain sequence of network descriptors.
But the order in which the descriptor are triggered by the GPU can be different.
"But the order in which the descriptor are triggered by the GPU can be different."
Does the problem which you mean lead to the NIC confusing about which WQE to send or which CQE to poll ? Because I want to use gds in a scene that multi-streams from multi-threads operate the only one SQ and SCQ, so the problem most concerns me is this issue.
yes triggering descriptor in the wrong order can confuse the NIC, e.g. GPU triggers desc n.2 before n.1 that would make the NIC to send both 1 and 2, and later when GPU triggers n.1 the NIC would be confused.
Thank you very much for your conclusion.
During these days I wrote a sample to verify this issue. GPU server will create some threads which send msg on different stream to the same SQ and poll the cqe from the matching SCQ. Client will wait for all replies from each server thread. Testing with this demo, I find that it can't work normally when more than two (not including two) threads or streams exist even though all threads are joined successfully. The server hang with getting not_ready_event of some streams with cuEventQuery and the client can't recv all notifiers from server threads.
The problem of client is caused by what you said I think. But I don't know what's the reason of server's hang with getting not_ready_event.
Activity
drossetti commentedon Apr 9, 2018
This is a very complex topic. The descriptors must be submitted to the CUDA streams in such a way that they will be triggered in the expected order, I.e. the same one in which they have been gotten back from the APIs.
Hannah-xxl commentedon Apr 10, 2018
Thank you for your answer. I wonder if using spin lock before exp post for SQ and unlock after getting the descriptors for SCQ can resolve this problem about accessing the same SQ and SCQ ?
I am not sure...
drossetti commentedon Apr 14, 2018
note that libgdsync APIs are not thread safe, due to the use of custom res_domain.
CPU locks would only protect the calls to IB verbs API and would result in a certain sequence of network descriptors.
But the order in which the descriptor are triggered by the GPU can be different.
Hannah-xxl commentedon Apr 20, 2018
"But the order in which the descriptor are triggered by the GPU can be different."
Does the problem which you mean lead to the NIC confusing about which WQE to send or which CQE to poll ? Because I want to use gds in a scene that multi-streams from multi-threads operate the only one SQ and SCQ, so the problem most concerns me is this issue.
drossetti commentedon Apr 23, 2018
yes triggering descriptor in the wrong order can confuse the NIC, e.g. GPU triggers desc n.2 before n.1 that would make the NIC to send both 1 and 2, and later when GPU triggers n.1 the NIC would be confused.
Hannah-xxl commentedon Apr 26, 2018
Thank you very much for your conclusion.
During these days I wrote a sample to verify this issue. GPU server will create some threads which send msg on different stream to the same SQ and poll the cqe from the matching SCQ. Client will wait for all replies from each server thread. Testing with this demo, I find that it can't work normally when more than two (not including two) threads or streams exist even though all threads are joined successfully. The server hang with getting not_ready_event of some streams with cuEventQuery and the client can't recv all notifiers from server threads.
The problem of client is caused by what you said I think. But I don't know what's the reason of server's hang with getting not_ready_event.