Starting Point: a Project with many Business Processes

In one of my last projects I worked in an environment with an application that had very many lengthy end to end (e2e) business processes. Technically, the system consisted of a framework that had been heavily configured and customized. Customization included changes to existing software and extending the system using new software.

The whole project setup was aligned to a v-model kind of progress, and there have been many companies involved working on the result, some onsite, some in India. There have been milestones and quality gates defined.

In a first project phase, the organization was clearly divided into customer and service provider and a fixed price contract was agreed with the service provider. From a QA perspective the customer had to provide a test infrastructure and had eventually accept the solution.  After each test level was a quality gate, all of them to be signed by the service provider but to be proved with suitable statistics. Just standard stuff for a v-model approach.

Although this looks very old fashioned, there have been some adaptations to the v-model to make sure the right system has been developed. The most outstanding was that rather early, the customer supplied a large number of business experts to support test as of system test, or even partially as of integration test. This way, basically the acceptance test ended up as just another regression test phase, only under different responsibility.

While I understand that many of the problems that we had with the project could be directly related to the overall setup, especially not involving test experts early enough, I would like to concentrate here on the aspect of e2e tests. With e2e meaning: business processes from the beginning to the end.

We had many e2e tests.

How come? Out of the many reasons I would like to point out the following:

• A very process oriented business scenario
• Strong involvement of business people and allowing them to set the frame and contents for the tests
• At least one external interface to a cash system that was very crucial to the whole system.

But not all of the tests were so great:

• Not conceptually designed, especially not checked for duplications
• Have grown into a mix of component or integration level checks and real business processes
• They tried to cover up for tests that had not been executed the test levels before
• They were missing out real potentially process based problems
• They were mainly oriented on happy path process flows

But for some new functionality there was a second phase with a chance to revamp the test process: Parallel to the proposal creation for this second phase, the business and engineering team pursued the preparation of the setup to enable a fast ramp up.

The project still was set up as a v-model shaped project, with teams at different locations. But the whole organization showed it learned and changed many things, including more early involvement of testers and end user business experts from the customer and the service provider at time of macro and micro design. The design was already pretty detailed and complete at the time, QA people were invited.

And what did we set up differently from a QA perspective?

As a consequence of the above situation, we decided to use the v-model in a more free way: the test levels and quality gates have not only been designed to fit the design levels, but also to make sure, that aspects resulting a testability analysis (Shift left as much as possible) were taken care of.

Use Process Analysis to Verify Design

In teams staffed with an end user business expert and a business analyst each (the latter with a bit of testing knowledge) we used mainly interview techniques und workshops in order to find out, which processes are most relevant for the business. During macro design, business processes already had been reflected, the new dimension of the QA activity was to verify if the macro design had relevant flaws. The resulting documents have been created mainly using pair programming techniques.

In order to enable this process, we defined a framework to follow. Upfront we had communicated the following objectives for the identification

• Cover most standard/relevant business processes from real life
• Cover most business processes designed in macro and micro design phase
• Use risk and complexity analysis to prioritize
• Have some tests to prove robustness and resilience of processes in case something is different than expected
• Somehow reflect roadmap of software to be developed

We also had the restriction that the resulting e2e tests were supposed to be readable and executable by business experts.

How to Achieve Coverage

The definition of coverage we used was different to standard code coverage measures. We used an iterative approach considering the available use case design from the macro design and the experience of the available business people.

• Starting with the “normal real life situations” of the business people, we would create a business scenario
• This was mapped against the design and validated for feasibility. At the same time, the covered requirements have been ticked off
• A next situation would be transformed to a scenario until the business people agree their main processes have been covered. This process was carefully controlled for not having duplications
• Starting from not yet covered design identify more business scenarios to be defined.

In reality this was a very iterative process, and in discussions between business experts and business analysts they supported each other in having ideas.

There also was one test person in the team. Her role was to enable the process, review the results and also to inject some negative thinking and assist others to use it: do not forget to design processes that include failures. Especially if these failures have an impact across different involved systems. And also sometimes to suggest to tweak a scenario in case it would help covering design features.

Result of this discussion was a set of almost not overlapping business scenarios to be tested, covering real life and design very well. Another result was a very good understanding that the design would work (plus quite some change requests ;-).

However, while looking at the design coverage, we ended up in a fruitful discussion: should we cover this bit of design in an e2e test? Or shouldn’t it be tested somewhere else?

Define Clearly Which Tests to Execute on Which Level

This really leads to my (anybody’s) mantra concerning the testing pyramid: do not create an ice-cream cone! Test in an e2e scenario only what has to be tested here – and make sure the rest is tested before!

To cope for this in the overall test concept there was quite some emphasis on defining, who is responsible for which test. Obviously everything should be tested as early as possible. But there are situations that can only be tested in an e2e situation.

Taking the above v-model,

1. the first quality gate would contain anything that could be tested within one component.
2. the second quality gate would contain all tests that can be performed within in one use case
3. so remains for the e2e test during system test phase only communications between use cases and validations that span across at least two use cases

Given a rather typical example: in one use case you have a screen wizard with several steps, building some data, that will fill a shopping cart for a customer. In a different use case you have some processing that that checks if a payment for this customer arrived. Then you have

1. screen functionality and validations (like “you cannot buy -1 items”) in component test
2. the integration test would be used to verify the complete order, for example if there are items in the cart and there is a person who ordered
3. in the system test we would then test an e2e process creating the contents of the cart, doing something in between (order processing) and then check the payment for this order before you allow delivery.

Sure, you would create the data situation for the payment checking use case already during integration test of that use case. But then you still do not know if the communication worked, especially considering the span of several use cases. Also, if you have validations that only make sense because of a certain order of execution use cases before, they are strong candidates for an e2e test.

Make Them Fly!

The first and most important result of the process analysis and test coverage analysis was sharing the understanding and knowledge with those groups who were responsible for unit and component test.

Next, we developed a data driven approach for execution out of the previously defined business scenarios. As the design was in a very advanced state we have been able to name all the screens, fields and buttons at least with logical names. We also included for test design purposes virtual names for result fields like error messages or result messages.

The overall process of cutting test cases was very generic: we used one test case per screen. Then we would combine those screens into flows using test suites.  For each business module in the application we used a tab in an excel sheet to define the screens potentially relevant for that module with all their screen fields and a filed for the error messages and for outcomes.

This excel book then has been used to define per business scenario all inputs and expected outputs. Together with test cases referring to these excel books, we had created a complete set of test cases with data and preconditions to be used for end to end processes.

Success?

This process of verifying the design, creating tests and sorting them onto the right level did include some learning curve. For our business experts it was a bit too technical, for our technical experts a bit too much business language. And all had to learn the process for creating test cases. But as the spread in the team was large, there was always someone knowledgeable available. In the end, a very productive approach was jointly shaped and test cases have been created very easily – after quite some up-front setup work that slowed down the process.

The structure of the test cases and their data driven approach supported the ability to use a keyword driven implementation in any test tool that supports key words (Like HPE ALM with UFT or BPT, or IBM Rational Suite). Together with a well-designed test execution strategy and test data strategy, automation would have been possible. The usability of these business process tests have been proven against the design. It could have worked. But as you see – I am using the conjunctive: unfortunately the proposal process was stopped by the customer to pursue other options, so we never had the change to provide the proof. So what do you think – would it have worked?