Repository navigation
UNCACHED memory allocation in Level Zero - Performance #515
Description
Activity
hi @jjfumero
we saw that the CACHED version is up to 4x faster than the UNCACHED
Correct. This is expected.
the UNCACHED flag can be used when buffers are streamed once, and not reused, so there is space in GPU's cache for other reusable buffers.
According to the specification, the flag means that the data would be cached or uncached. Currently, that means that when programming a command containing this allocation, and we dont cache those allocations if UNCACHED has been set. Of course, as you say, if we dont cache those buffers, then the cache would have space to cache cacheable buffers, which is expected.
could you elaborate on what documentation you think it is missing?
Hi @jandres742,
thanks for the clarification.I meant perhaps something like this:
diff --git a/level_zero/core/test/black_box_tests/zello_world_gpu.cpp b/level_zero/core/test/black_box_tests/zello_world_gpu.cpp index c61455610..9b7c6ec37 100644 --- a/level_zero/core/test/black_box_tests/zello_world_gpu.cpp +++ b/level_zero/core/test/black_box_tests/zello_world_gpu.cpp @@ -24,6 +24,12 @@ void executeGpuKernelAndValidate(ze_context_handle_t context, ze_device_handle_t // Create two shared buffers constexpr size_t allocSize = 4096; ze_device_mem_alloc_desc_t deviceDesc = {ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC}; + + // Set the flag to UNCACHED buffer. Note that this flag is used when streamed once + // to the accelerator (the GPU in this case) and data is not reused. This optimizes + // the use of cache for other reusable buffers. However, when using the `UNCACHED` + // flag, GPU kernels are expected to run up to 4x slower than the CACHED version. + // (ZE_DEVICE_MEM_ALLOC_FLAG_BIAS_CACHED) deviceDesc.flags = ZE_DEVICE_MEM_ALLOC_FLAG_BIAS_UNCACHED; deviceDesc.ordinal = 0;
I noticed that all the tests and examples provided here https://github.com/intel/compute-runtime/tree/master/level_zero/core/test/black_box_tests uses the
UNCACHEDversion without any warning about performance when invoking kernels multiple times. IMO, better documentation will help developers to be aware of the decisions made to set flags in Level Zero (which are a lot).It would be good as well to have some information about in which situations the
UNCACHEDversion might be used in the Level Zero spec.If you agree, I can prepare a PR with this documentation.
thanks @jjfumero . For the bb tests. Yes, we have the UNCACHED there. There hasn't been any special reason behind it, other than maybe we added it in one test and then we just used that test as base when creating new ones. I guess also we hadn't thought about it since the BB tests are meant to show how to use the APIs, not for performance.
Now, we could add the documentation to the spec, indicating that when using UNCACHED you might see performance drop. But, wouldn't be that expected, that when you dont cache your data, and are constantly using it in the GPU, then you have a larger latency since you need to read from a farther level in the memory hierarchy? This is true not only for GPUs, but also for CPU development.
This issue is a suggestion based on our findings.
Some of the examples in Level Zero are using the
UNCACHEDFlag : https://spec.oneapi.io/level-zero/latest/core/api.html#_CPPv438ZE_DEVICE_MEM_ALLOC_FLAG_BIAS_UNCACHEDFor example here:
https://github.com/intel/compute-runtime/blob/master/level_zero/core/test/black_box_tests/zello_timestamp.cpp#L102
In TornadoVM we have run a lot of experiments with
CACHEDandUNCACHEDand we saw that theCACHEDversion is up to 4x faster than theUNCACHED. As I understand, theUNCACHEDflag can be used when buffers are streamed once, and not reused, so there is space in GPU's cache for other reusable buffers. Unfortunately, the Level Zero documentation does not warn about this. From our experience, this is very "error prune" since we were analyzing the number of threads and block of threads deployed, rather than how memory was allocated.Does it make sense to add some documentation in the Level Zero examples to include this information? As well as in which situations developers may use the
UNCACHEDvsCACHEDflag?BTW, I am not sure if this is the right repo to file this issue or I should also open one in Level Zero.