LTP results are sent to KCIDB under multiple names

[spbnick]

KernelCI native LTP results are sent under multiple names to KCIDB. An example revision has five instances of LTP reported, all under different names (ltp-timers, ltp-pty, ltp-mm, ltp-ipc, and ltp-fcntl-locktests):

See the attached revision data for details.

This violates the agreed-upon test naming scheme, makes it difficult to see complete LTP results (you have to check all the names), and impairs correlation with other submitters (which might use different names, or just the correct single name).

E.g. ARM submissions use a single name:

Underneath which all the subtests are placed together:

Apparently the results use different names, in part because these are executions of different subsets of LTP, split apart for faster parallel execution. However, from KCIDB POV it doesn't matter how many tests of a suite were executed, the test suite should still be reported under its name.

Please consider sending results of each test, including LTP, under its "canonical" name, regardless of which subset of its subtests is executed.

[spbnick]

@mgalka promised to take a look at this 😁

[gctucker]

How should the test cases inside the test suite be named? Is there a convention for that too?

[spbnick]

How should the test cases inside the test suite be named? Is there a convention for that too?

That's entirely up to the test suite. We can't really control that.

[gctucker]

OK, then we still won't be able to actually compare results between the different data sources other than manual inspection.

[spbnick]

OK, then we still won't be able to actually compare results between the different data sources other than manual inspection.

Hmm, why? Doesn't LTP generally keep the test names the same?

[spbnick]

Could you describe how you'd like the results compared?

[gctucker]

If individual test cases coming from different CI systems don't have the same names then they can't be automatically compared.

Take this one for example:
https://staging.kernelci.org/test/plan/id/61a5197941746c4e383d35b4/

The full test case names are made up because each test is run several times as part of the test suite, when there are several media interfaces. It wasn't really designed to be automated so the test case names can't be used directly as they're found in the log.

For LTP, I believe test cases have a more controlled naming based on the executable. Still, what if they're part of a subset? Take this one for example:
https://staging.kernelci.org/test/plan/id/61a503cf0732cfa3d23d35e4/

Should the test names be like ltp-mm.ksm01, or mm.ksm01, or ksm01? If there's nothing enforcing that in KCIDB then anything can happen.

[spbnick]

The full test case names are made up because each test is run several times as part of the test suite, when there are several media interfaces. It wasn't really designed to be automated so the test case names can't be used directly as they're found in the log.

Well, I'm not sure I understand what's happening there completely, I would need to know more. However, either it's OK (e.g. different names describe a meaningful test change), or we need to talk to the test maintainers about making test names stable and putting the instance information somewhere else, or maybe we could apply a simple (and reliable) transformation to clean them up.

For LTP, I believe test cases have a more controlled naming based on the executable. Still, what if they're part of a subset? Take this one for example:
https://staging.kernelci.org/test/plan/id/61a503cf0732cfa3d23d35e4/

That's exactly the situation I'm talking about. Even if you're running a subset of LTP tests, their test paths should still start with ltp. After all, we don't rename ltp when a test is added to, or removed from it, right?

I.e. it should look like this:

However, perhaps that name is required to make the KernelCI framework work, and that's OK. However, those names should be transformed before submitting to KCIDB then.

[gctucker]

So basically, KCIDB doesn't specify exactly the test names but the guidelines are to not mangle at all the test names when they're clearly defined by the test suite (e.g. just the executable name for LTP) and coordinate with other CI systems when the test names aren't clearly defined by the test suite such as for v4l2-compliance.

[spbnick]

That's exactly the situation I'm talking about. Even if you're running a subset of LTP tests, their test paths should still start with ltp. After all, we don't rename ltp when a test is added to, or removed from it, right?

Just to make sure I got it right, each of those ltp-timers, ltp-mm, ltp-ipc, and so on, is just a different invocation of LTP, with different parameters, and is not an invocation of a part, e.g. a separate program that is contained within LTP?

[gctucker]

Yes they are just a subset of the LTP tests being run. For example, ltp-ipc is running all the ipc tests from LTP.

[spbnick]

So basically, KCIDB doesn't specify exactly the test names but the guidelines are to not mangle at all the test names when they're clearly defined by the test suite (e.g. just the executable name for LTP) and coordinate with other CI systems when the test names aren't clearly defined by the test suite such as for v4l2-compliance.

Yep. That's what we could do with the situation realistically, I think.

[spbnick]

Yes they are just a subset of the LTP tests being run. For example, ltp-ipc is running all the ipc tests from LTP.

Well, then that's what I expected it to be, and in KCIDB their paths should start with ltp.. Otherwise, we can't really correlate them.

[spbnick]

So basically, KCIDB doesn't specify exactly the test names but the guidelines are to not mangle at all the test names when they're clearly defined by the test suite (e.g. just the executable name for LTP) and coordinate with other CI systems when the test names aren't clearly defined by the test suite such as for v4l2-compliance.

And coordinate with the test suite authors, of course, if possible.