[UR][CTS] Upstream CTS parser

[veselypeta]

Credit: @martygrant

We had used this parser script to measure the CTS pass rate of each adapter in our internal CI. Since UR holds all the CTS tests it makes sense to upstream it here.

I've modified martin's original parser such that we run conformance test suites individually. Instead of parsing the output from stdout we can use the GTEST_OUTPUT environment variable to specify that GTEST should output it's testing summary into a json file. We can then parse this and summarize the results.

Example output:

[martygrant]

Hey Petr this looks good, wasn't able to make changes myself last week. Would this still have the issue around crashed tests? The total number of tests wouldn't be accurate because any failed suites won't produce an output file?

[veselypeta]

Would this still have the issue around crashed tests? The total number of tests wouldn't be accurate because any failed suites won't produce an output file?

Yes this is still an issue as you say that the output file is not generate. However I think long-term we shouldn't expect there test suites to fail and obtain an accurate value from all the test suites. See discussion here #817 (comment)

[pbalcer]

Would this still have the issue around crashed tests? The total number of tests wouldn't be accurate because any failed suites won't produce an output file?

Yes this is still an issue as you say that the output file is not generate. However I think long-term we shouldn't expect there test suites to fail and obtain an accurate value from all the test suites. See discussion here #817 (comment)

One thing we could do is to first look at the list of gtests in a binary, add them to some set, and remove them if found in the log. When summarizing, we can count the tests that remain in the set as Unknown/Crashed.

[veselypeta]

I now use gtest_list_tests to get the test list. If a test suite crashes we just assume that all the tests fail. We could get more granular with this by running each test individually with Popen but I think this is unnecessary.

Example output: