Reporting DSL

Architecture

The Reporting DSL is based on the possibility to inject a reporting client into the main test code with just a few annotations. The client can then be used to report progress of the test.

As opposed to the concept of remote execution where proxy agents can be used to actually call systems under test, here the reporting client is directly in the runtime where tests are run, usually pipelines or localhost. So that runtime needs to have access to the testing tool directly.

Reporting heavily depends on the tool used for test catalogue and test run reporting. Today Pumpo#5 uses an abstraction that might be extended or modified as new tools are being added to the list.

A key part is to configure mapping of the test to the right test case. This is done using the annotation @ReportTo with value set to the test case id.

There are two main possibilities how to use the reporting DSL. One that is easier to implement is to add the reporting client as another parameter of the test method. The parameter should be the interface Reporter from Pumpo#5 or extend that interface. Then it is possible to use methods provided by that interface.

Example:

@Capability(key = "browserName", value = "chrome")
public interface MySimpleWebApp extends WebApplication {
    default LoginPage openLoginPage() {
        return this.open(LoginPage.class);
    }
}

@Navigate("url.com/login")
public interface LoginPage {

    @SetValue("email")
    LoginPage fillOutEmail(String email);

    @Click("loginButton")
    LoginPage clickLogin();
    
    @AssertElementContent("loggedInDiv")
    LoginPage verifyThatLoggedIn(String expectedText);

}

class TestClass {

    @Test
    @ReportTo("TA-305")
    public void sampleTest(
        @Capability(key = "browserName", value = "pn5-chrome")
        MySimpleWebApp myApp,
        Reporter reporter) {

        reporter.startTestcase();
        LoginPage loginPage = myApp.openLoginPage();
        reporter.completeStepPassed();
        loginPage.fillOutEmail("test@test.com");
        reporter.completeStepPassed;
        loginPage.clickLogin();
        reporter.completeStepPassed;
        loginPage.verifyThatLoggedIn("Your are logged in!");
        reporter.completeTestcasePassed();

    }
}

On the background Pumpo#5 will initialise the reporter client also based on necessary configuration provided in configuration files and will report progress of the test to the right test case (id TA-305 in this case) in the right test plan / pack and in case there is any failure (exception) then it will automatically report failure with additional details added to the test step where the failure occurred.

The issue with the previous example is that the fluent interface does not work anymore every time we need to report the success of a test step. There are additional types to specify in order to comply with Java syntax. To overcome that issue there is a more complex possibility how to use the Reporter interface, this is by extending the Reportable<> interface in all objects where reporting will be used:

@Capability(key = "browserName", value = "chrome")
public interface MyWebApp extends WebApplication, Reportable<MyWebApp> {
    default LoginPage openLoginPage() {
            return this.open(LoginPage.class);
    }
}

@Navigate("url.com/login")
public interface LoginPage extends Reportable<LoginPage> {

    @SetValue("email")
    LoginPage fillOutEmail(String email);

    @Click("loginButton")
    LoginPage clickLogin();
    
    @AssertElementContent("loggedInDiv")
    LoginPage verifyThatLoggedIn(String expectedText);

}

The flow of the test then becomes:

class TestClass {
@Test
    @ReportTo("TA-305")
    public void sampleTest(
        @Capability(key = "browserName", value = "pn5-chrome") 
        MyWebApp myApp, 
        Reporter reporter) {
    
        myApp
            .report().startTestcase()
            .openLoginPage()
            .report().completeStepPassed()
            .fillOutEmail("test@test.com")
            .report().completeStepPassed()
            .loginPage.clickLogin()
            .report().completeStepPassed()
            .loginPage.verifyThatLoggedIn("Your are logged in!")
            .report().completeTestcasePassed();

    }
}

Reportable<MyClass> uses the Java "generics" where MyClass is the type to be returned once a method of the interface Reporter is used (startTestcase, completeStepPassed, etc...).

Both approaches produce the same result.

Method reference

Reporter::startTestcase

Starts the test case that is passed to the test using the annotation @ReportTo by setting it as in progress.

Reporter::setResult

Parameter	Type	Description
result	TestcaseResult	Result of the test case

Sets the result of the test case to specific value of the enum of type TestcaseResult, that is PASSED_UNTIL_FAILURE, PASSED or FAILED. The default is the first value but under some conditions it may be useful to already set the final result while continuing reporting at step level.

Reporter::getResult

Returns the result of the test as enum of type TestcaseResult.

Reporter::getException

Returns the Throwable that has been previously recorded as the cause of the test case failure. Returns null if no such Throwable was set.

Reporter::setStepFailureBehaviour

Parameter	Type	Description
behaviour	StepFailureBehaviour	Intended behaviour

Configures the current step failure behaviour to one fo the values of the enum of type StepFailureBehaviour, that is: STOP_AS_FAILED, CONTINUE_AS_PASSED_UNTIL_FAILURE, CONTINUE_AS_FAILED or CONTINUE_AS_PASSED. The first one is the default and will cause the test to halt if a Throwable is thrown. In case the test should continue the other values allow setting what should be the result captured for the whole test case.

Note: Calling this method alone with a value such as CONTINUE_AS_FAILED is not sufficient to continue the test in case an exception is thrown. The exception needs to be caught and passed to the method completeStepFailed that will use the intended step failure behaviour to decide whether the exception needs to be rethrown or swallowed to allow continuing the test flow. Instead of manually catching the exception and passing it to completeStepFailed we advise to use the more concise method doAsSingleStep that allows to do all in one.

Reporter::setStepFailureBehaviourDefault

Parameter	Type	Description
behaviour	StepFailureBehaviour	Intended behaviour

Configures the default step failure behaviour for next steps. The current step failure behaviour is not affected. See setStepFailureBehaviour for details how is the value used.

Reporter::skipStep()

Marks the current step as skipped. The status of the step might differ based on the actual implementation, e.g. instead of skipped the value might be todo.

Reporter::skipStep(String)

Parameter	Type	Description
comment	String	Comment for the current step

Adds a comment to the current step and marks it as skipped. The status of the step might differ based on the actual implementation, e.g. instead of skipped the value might be todo.

Reporter::commentStep

Parameter	Type	Description
comment	String	Comment for the current step

Adds a comment to the current step without any other action. Based on the specific implementation any previous comment might be replaced if the reporting only allows one comment per step.