Learning to be a Tester: Automation Testing
in previous post we went through the basics of software testing and now lets explore the automation testing. Functional testing is required to be repeated over and over with every new commit or change in the system. For an application with considerable features this could be a very lengthy and cumbersome process. Automating the functional tests saves time and provides frequent effective feedback.
There are various tools and frameworks used for automation. Lets look at first various types and level of automated tests. The best way to make sure that your test automation is going in the right direction is by following the automation pyramid conceptualized by Mike Cohn, I will be sharing it in the context of how we use it at Amplify.
We have majorly three set of tests to test the functionality as displayed in the automation pyramid below:
Note: There could be other types of automated tests to test non functional aspects like Performance. Here we are only talking about functional tests.
Lets look at various types of automation tests that we have in our pyramid:
Unit Tests: Kent Beck introduced TDDto the programming world that relies on writing an automated test first and then writing minimum code to pass this test. These tests that test a small piece of code( a functional unit of code example - class ) are unit tests. Since unit tests are written before the actual application code they are mostly written and maintained by developers. These are the first gate in the system and are run as part of automated builds after each commit or code change. Unit tests as they don’t go through the actual UI are very fast and reliable, therefore they should be the highest percentage of automation tests in a system as shown in the diagram.
Tools: Unit tests are written using a test frame work compatible with the programming language used for the application code example: Junit is used for Java, Rspec for Ruby, etc.
Contract Tests: The way unit tests test a class contract tests is a test for an interface. Writing contract tests is often referred to as contract programming or Design by Contract. Contract tests metaphor comes from business contracts where a consumer and a supplier enter into a contract around benefit and obligations. Similarly in reference to programming a contract test ensures that the contract between a service provider and consumer is always intact. Since the contract tests verifies the integration points in a system their percentage is less than the unit tests and thus sit at the middle level of automation pyramid. They are also run every time a change is made on supplier side but maintained are by consumer.
Tools: Contract tests can also be written in a similar framework as unit tests or an acceptance test framework like Cucumber.
UI Tests: User Interface tests go through the actual UI of the system and simulate the user actions. These are often the most critical user flows from the regression scenarios for a system. Since these tests simulate the user actions they are highly unreliable and brittle therefore these are the smallest percentage of tests that a system has. UI tests are also run as part of the automated builds providing continuous feedback but they are the slowest running tests. There are ways to improve speed and reliability of these tests using better tools and techniques which depend on type of application and tool used. UI tests are written and maintained by both developers and QA team preferably in a collaborative manner.
Tools: Most widely used UI testing tool for web based applications is Selenium webdriver, others are Watir, Cucumber, Sikuli, FitNesse. Whereas for native and mobile apps tools available are: Appium, Calabash, UIAutomator for Android, UIautomation for ios.
Note: At Amplify we have written a library called HoneyDew that enables us to write UIAutomator tests using cucumber.
Often teams add UI tests to their automation suite on a regular basis and after a certain period with the growing functionality this suite explodes. An automation suite that takes hours to run is not a good feedback source. They end up not maintained and to a point where they become useless. Big or small all teams should always follow the automation pyramid to have continuous feedback.