For this I was thinking the following API:

At the low-level there should an xfail() test method very similar to the skip() that will raise a specific exception denoting the expected failure with a custom message. This will be handled from the runtime similarly to the way the skip exception is handled: print an "XFAIL" note instead of "OK" and add the user message.

The xfail() method would work nicely for sanity checking, as it could be called from within the @sanity_function, but this is not the case for performance variables, which are checked against their references at a later point by the framework. For this reason, we should also have a builtin @xfail decorator. This could also be used to decorate a @sanity_function.

This @xfail decorator should come in the following two forms:

@xfail("message", when={'var': 'value', ...})
@xfail("message", when=lambda t: t.var == 'value')

In the first form, we can support some pseudo variables that we expect to be used often, such as $sysenv to translate to the system/environment combination of the test etc.

In the second form, the when argument is a callable that will be called by reframe with the test instance as its argument and should return True or False to determine whether the test is an expected failure or not.

Often performance variables are defined explicitly by setting the perf_variables variable without using the @performance_function decorator. In this case, the following construct could be used once the perf_variables dictionary is populated:

builtins.xfail("message", when={'var': 1})(self.perf_variables['foo'])

This essentially applies explicitly the decorator to the performance function stored in a specific "perf variable". When using the xfail builtin like this, the when argument could be replaced with a tautology and the condition check moved outside:

if self.var == 1:
    builtins.xfail("message", when=lambda _: True)(self.perf_variables('foo'])

Support marking a test as an expected failure #2378

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions