Every software or code has to be tested, including those of SAP BW users. Especially if you want to work in an agile mode, providing high quality consistently is crucial. Too often, SAP BW users only use the graphical user interface (GUI) as the preferred testing tool. On the one hand because it is simple to use and on the other because there were hardly any alternative approaches available. Unfortunately, GUI tests are always End 2 End (E2E) tests and never lead to success alone, but in inadequate runtimes and insufficient coverage.
In this article, we explain the reasons why that is, which additional testing methods should be used and what improved testing strategy results out of it. In the end, the reader will know how to efficiently test SAP BW Transformations, Queries, HANA Calculation Views, Procedures to become Scrum or Dev-Ops ready.
The limits of GUI based E2E testing
One thing is clear, there is no Swiss Army knife in software technology or in testing, or a solution that serves everything. Neither can you succeed with GUI tests that always test the entire software stack End2 End. There are several reasons for this, but let’s list the biggest ones:
- it requires cost intensive test data management
- it is too slow and too expensive (not efficient), because it is more tested and covered in a test case than required
- too many system dependencies, which is why test cases quickly invalidate
- no comprehensive test design is possible (not effective)
- error investigation on the whole stack and real data is very costly and time consuming
- can only be done on to the test /quality system which is time consuming
This test method is probably so much used, because there is a fancy feature: screen recording, which accelerates the speed of test case creation and seems to be quite intuitive in its usage. But one day you will reach the point where solely GUI testing becomes just too expensive, too slow and inefficient. Despite immense efforts, the tests do not run overnight or weekend and you cannot guarantee the quality and delivery dates anymore. Because of that, GUI tests are part of the integration testing and serve, among other things:
- to test the interplay of the interfaces of objects / modules
- validate process flows (e.g. process chains)
- validate run times
At the end this strategy leads into the Ice Cream Anti-Pattern, in which the expenses are distributed the wrong way around:
Almost everyone got to this point
At SAP Tech-Ed we were often asked what can be done in such a case, if you have reached exactly these limits of GUI testing. Our answer: flip the efforts and use more Automated Component Tests and less GUI based integration / E2E tests. It was these customers who got to the point to recognize that they tried to solve the big picture using a single tool.
Quality assurance requires Component Testing
The solution to the above explained issues is called Component Testing (aka Automates Unit Testing). Within Component Testing an object is isolated in a way that it can be tested independent of the system environment or any other dependent object (e.g.: Dataprovider, Database Table, Calculation View, Function Module, etc.). This is nothing new, many industries work according to this scheme. A car will not be tested on the street for the first time and a laptop will not run the first time after its final assembling. This final “big-bang” integration test an innumerable series of Component Tests are executed in precedence, in which each element is individually isolated and independently validated from the rest. Only if all components have successfully completed all Component Tests, the actual integration tests are performed as an interaction of all (sub) components.
In software Component Testing, parts of the software are tested separately from the rest. It focuses entirely on the validation of the functionality of an ABAP class, SAP BW Transformation, Query, HANA Calculation View, Procedure, etc. This is not simple to do for SAP users, but the benefits are immense:
- no “test data management” is required
- testing is already done on the development system because AUTs are independent from the system’s data and other objects (which make them regression tests) which results in very stable tests cases and very fast testing (no transport to the quality system is required anymore)
- results are constant and independent of system state / variables which is also important for regression testing
- a minimal number of just a few test data rows are sufficient (efficient and perfect for debugging)
- test design reduces the number of test cases to the required minimum (efficient)
- everyone can test in parallel to others
That’s why we’ve made it our mission to make Component Testing available to SAP BW users without any developer knowledge, so even the department can support here. This creates many more new benefits, such as the elimination of shadow testing and a high level of confidence in quality assurance on business side. Finger pointing will not exist anymore. But the most important aspect is that a completely different test strategy is feasible now.
The optimal test strategy for SAP BW / HANA
In our opinion, the optimal test strategy is based on the test pyramid (first introduced by Mike Cohen), which states that the tools and efforts should be used according the best cost / benefit ratio. For proving functional correctness there are basically: AUTs, integration tests, and manual tests. Most of the functionality, at 70–80%, should be tested via AUTs, the rest through integration testing, user acceptance tests and manual testing (for example what is too difficult or costly to automate can remain as a manual test).
By shifting the efforts, the initial problem, why tests in the SAP BW environment break any deadline and exceed the budget, is resolved though limiting GUI integration tests to its purpose only. Fully automated coverage of all test cases can be achieved, which is the foundation for Dev-Ops and Scrum.
A Scrum cycle without a high test automation, will end up for a SAP BW team in testing only, or no more testing, or only limited testing — probably the most likely scenario.
This also has positive effects on working in the SAP landscape. Most of the testing is done in the development system, without the need for transportation. This additionally accelerates the development cycles. Errors from the production can easily be simulated and fixed in the development system via AUTs, so they are automated and checked forever.
Due to the high granularity of testing, Component Tests can be directly linked to requirements of the business department, for example from JIRA or HP-QC. The exciting thing about it is that a requirement becomes an AUT and the logic is perfectly documented that way, something we call an “automated documentation”.
Imagine, external consultants or employees leave the project and their developments are fully secured and documented by automated Component Tests. This will make the difference whether you sweat on later changes or just press a button to validate in seconds that all former requirements are yet met.
Summary
Excessive End 2 End Testing inevitably leads to the anti-pattern of the ice cream cone. Test runs take too long, are too costly and agility
and quality of the developments is no longer guaranteed. Full and optimized test coverage can not be achieved this way.
According to the test pyramid, test expenditures should primarily be shifted to lower levels. Due to a focus on fully automatic component tests (aka unit tests)
the mentioned issues with E2E testing only can be mitigated and an optimal test coverage can be achieved.
With our solution xGile its possible to create AUTs
without programming via a unified user interface for SAP BW Transformations, Queries, HANA Calculation Views and Procedures.
An efficient and complete test coverage in SAP BW is possible!
