For those of you who don’t know what we do, we are building an education technology platform,…
Learning to be a tester!
Recently I was searching good material that I can share with my team mates who have just taken up the role of a QA. Although there are hundreds of posts out there but I couldn’t find one concise article that would explain the job of a QA professional in an agile project. Here is the synopsis of what I think would be a first stepping stone of a QA.
As I explained in my previous post ‘Choosing to be a tester’, the responsibility of a QA is to assure quality of the whole system. A QA would achieve this by performing various tests on the system at various stages. Lets talk about types of testing that can be done on a software or a product. I will explain this by taking a non software example. I have been asked this questions many times and this is my favorite interview question as well.
'How would you test a pen?' or any random object, lets list out all the types of testing and various scenarios that can be performed to test our pen:
1. Functional testing: it is a type of black box testing that bases its test cases on the specifications of the software component under test. For a pen the main specification would be that ‘as a user I should be able to write using pen on a paper’. We would test the pen by writing on a piece of a paper.
2. System testing : is the testing conducted on a complete, integrated system to evaluate the system’s compliance with its specified requirements. Another specification for our pen where we need to perform system testing would be ‘as a user I should be able to put cap on my pen’. The test scenario here would be test placing the cap on the pen and making sure its tightly fasten and doesn’t comes out when pen is dropped or thrown away.
3. Stress testing : is a form of deliberately intense or thorough testing used to determine the stability of a given system or entity. The user scenario here to test would be ‘as a user I should be able to use pen in rough conditions to a certain limit’. As part of stress testing we are testing the extreme conditions under which our pen would continue to work, like: on a rough surface: wooden floor or granite floor, using pen with extreme pressure and other intense conditions.
4. Usability testing : is a technique used in user-centered interaction design to evaluate a product by testing it on users. This is an important testing from a users perspective. Since the user of a pen can be from different age group with different personality, it is important that it meets the requirement of a majority of the group. The main scenarios to verify would be: assuring the pen fits hands of a 5 yr old to 50 yr old person, both left and right handed people should be able to write with it, its grip should be comfortable for long usage.
Above were few relevant testing types to be performed on a pen, there are various other testing types that would come in picture for another product.
When talking in reference to software above testing types can be done manually and automatically. I will go over manual and automation testing in my next post.
Till then enjoy testing!!!!
In an Ideal software development world there would be no QA team or a testing team, but we all know all the processes including software development are prone to human error. With an increasing demand for better software there is a rise in Quality Assurance jobs.
But most of the business or stake holders consider the QA team as an overhead and whenever their is a slump period the axe falls first on the testers. The development team as well often finds it difficult to fit in the testing team into their development phase and the teams usually end up on two poles of the process. And I can imagine the poor Project manager at the centre somehow managing and trying to get work done to get his billing correct. Facebook doesn’t have any software testing team but they do as well test their software through beta testers before launching a feature to the world.
How and where exactly does the testing team fits and can they really improve the software industry
Well the answer is difficult but certainly not impossible. Its not the business or the developers who need to change as they are not necessarily wrong. Mostly its the test engineers who can change this attitude, what’s needed is some retrospection into the QA role.
Given that we are still not into the 100% agile mode most of the testing teams in traditional waterfall model work in post development phase. Where the opportunity of change is almost zero and even the defects fixes turns out to be long process. Testing teams are often divided into manual and automation teams.
- Manual team who consider there job limited to writing manual tests and executing those tests. These tests often run into thousands and updating them is again a huge effort. At times these tests are written years ago and the person executing them hardly understands the business goal behind a given step. More the number of bugs they find is a metric of how well the job was done.
Our client once said why does your team total bug found was always less, the QA team doesn’t seems to be working well? The team was working way better as we had zero bugs in production, isn’t that the ultimate target?
- Automation teams on the other hand don’t care at all about what’s in the test cases they simply automate them line by line. Their performance is measured on the number of test cases automated (you can leave the test script “to do” to improve your performance, your manager is never going to open the test case and verify it? Yeah many people actually do that !!)
What’s the problem with above structure which is pretty much followed by more than 50% of the software teams?
Wiki defines Software Testing as “Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test”
The ultimate aim of testing is to provide feedback on the software to the stakeholders before its being released to the end user. The problem with above stated structure is the missing feedback, both the teams are trying to get their job done but no one is focussed on providing quality feedback.
Let’s see how we as QAs can get this whole cycle right. Testing team is the representative of business to the development team and vice a versa as they get their hands dirty on the software before anyone else and continue to do that through the life of the project. They often know more about the functionality then anyone else if not then they are probably not doing their job right. With my experience below are the certain points that seemed to be working and can help anyone who wants to improve upon:
- I haven’t seen the concept of different manual and automation teams working but given the limited availability of required skill set this model is the only choice in certain cases. But even with this model both the team should work more closely to get the automation priorities right and manual team should use the feedback from automation results in manual verifications. A testeer who has understanding of business can definitely test extensively and write better automation tests.
- Testing team should work with Product team to understand the business goals and be more flexible in their manual testing to incorporate the business aspect.
- The projects where testing teams work closely with development team are more successful as the feedback loop for any defect/ bug is shorter. Even for offshore models the development and testing team for a given module should work next to each other.
- There is no way the performance is proportional to the number of bugs found, seriously? This is the biggest myth of testing world. The job of QA is to uncover issues in a software but higher the number of bugs higher is the cost. QA should discuss the edge cases with developers before building the functionality or raise the risky area ahead of time to the product managers so that the bugs can be avoided as early as possible.
The above points are very much useful and applicable to agile projects as well. Given agile testers are not born they are the same testers who have worked in the waterfall model at some point of there life as a tester like me.
Estimation and testing
After more than a decade since Agile Manifesto was published there is a wide spread adoption of agile practices in software industry. Although it had been more prevalent in small and start up companies but in last 2-3 years even banks and big companies have been learning the agile ways. With this increase in agile popularity there is a surge in failures attributed to agile methodology. As part of a research it was established that how unsuccessful Agile has proven to be, the key points from the report are available here. But there are reason for these failures, a great article "How To Fail With Agile:" explains possible causes behind these failures and how teams can avoid these pit falls.
Being a QA on various agile teams I have observed that even for successful agile projects, teams find it difficult to extend agile practices to the test/QA teams. These teams usually have their dev teams following agile but their QA team will still be in waterfall. The reason being, QAs are not able to catch up with the dev team and are lagging behind to cover up with automation and manual testing.
One of the important reasons for this gap is the way teams do estimation. All agile teams follow story or feature estimation as part of their planning process. There are varied number of ways that one can do estimation which can represent either complexity or time. In scrum methodology estimation activity is called planning poker and usually the stories are estimated in fibonacci numbers series : 1, 2, 4, 8…. or they could be S, M, L, XL sizes. During iteration planning meetings (IPM) developers throw these numbers for a given story to get estimation.
But a story other than development work involves testing phase which could be unit tests, acceptance tests, exploratory tests and regression tests. Since developers write unit and acceptance tests they are included in the estimates but the work done by QAs is not considered in these estimates. There could be argument that testing work can be done in parallel with development activities and hence we need not allocate extra time for this. But as part of TDD and BDD developers also share the responsibility of writing tests and setting up test environments. Then there are other testing processes that involves exploratory testing, setting up test data, fixing automation tests and other testing activities that vary with each story. Some times there are stories where QA effort might be more that dev effort.
The best way to make sure that QA teams don’t end up being bottleneck for releases they should be included in planning meetings and should be given power to vote & estimate QA effort involved in stories. The QA team might not be bale to contribute much initially but as they get experienced in the estimation process the would be able to get accurate estimates. The QA estimates need not be added to dev estimates instead they should be averaged out and you will get a true velocity of the team.
Effective BDD with Cucumber
After attending couple of events in last few months I came across the reality of our BDD world. Most of the teams that use cucumber or any other BDD tool have adopted them as QA tools. There developers or business or product people never look at these tests and are sole responsibility of QA team. Does this sounds familiar? do you do it the same way? is cucumber killing you?
BDD is an evolution of TDD and by definition it aims to help focus development on the delivery of prioritized, verifiable business value by providing a common vocabulary that spans the divide between Business and Technology. There are various tools and frameworks that can be used to implement BDD and cucumber is by far the most used.
Before you decide that cucumber isn’t the right option for you, lets analyze different reasons that make using cucumber less effective:
- Test written by QA team only: The biggest advantage of BDD is in enforcing more collaboration amongst different roles in projects. When developers are not writing a single feature test, then there is no behavior drive development and the collaboration is not achieved. The tests in this scenario will be regression tests and not inline with the system design.
- Test written only by Developers: Again we loose the collaboration aspect here but what we lose more is the user intent. QAs bring the user aspect to software testing, with QAs input we can avoid writing scenario that end user will never face and hav better user experience implemented in our system.
- Product Owners: In an ideal scenario in BDD world we expect BAs or Product managers to write feature tests with QAs. But I haven’t seen this model being successful anywhere. I haven’t seen any product manager buying this idea ever. The best option to get the business intent into our tests is to have clear deliverables and acceptance criteria in our user stories. These can then be translated into feature tests by QAs and developers.
- Regression suite: Cucumber tests are very valuable in providing feedback and they majorly serve the purpose of testing various integration points in your system. Using them as regression suite by testing all possible negative use cases will not be fruitful. The well known test automation pyramid recommends the amount of integration tests should be around 5-15 % of entire tests and this applies to cucumber tests as well.
- Java script testing: With the ever rising popularity of selenium-webdriver the amount of java script testing in cucumber testing is also very much in practice. I have even seen team duplicating there entire tests to run against IE. Just because the tool supports these features doesn’t guarantees return if used at large scale. Java scripts test in selenium have been historically brittle and will continue to be. The effective way is to use a Java script testing framework like Jasmine and few end to end tests in Selenium to test the UI.
- Test Case documentation: Whilewriting BDD tests we don’t add comments to our tests as the feature tests serves the purpose of documentation. Doesn’t this applies to your manual test cases? Teams use test case management systems to document there test cases, with BDD you should get rid of that as well. Cucumber features are very effective way to manage test cases that are regularly updated without any extra effort.
BDD is a great practice to make your development cycle effective and cucumber is an amazing framework for writing tests and collaboration. You just need to do it right.
Choosing to be a tester!!
During interviews I have always come across candidates who wanted to be developers but landed up in QA somehow or they started with QA considering it to be an easy job and want to move to development as they gain experience. I have always been supporter of changing profiles, trying different roles but this one not so much. As those I have seen taking this path are mostly doing this because either they think developers have better roles then testers or they think development is too difficult. According to me if you are working as a tester and believe in any of these two then you probably don’t know your art well enough.
I didn’t chose being a tester or would say that was the only road I had seen in my initial career. And I am happy that it out to be the right choice. But what about those who have choice? Given that the average developer to tester ratio is around 4-5 developers per tester, there are good number of testing jobs in the market. And the testing community is definitely in need of people who are passionate about their work. I am highlighting some points that I like most about being a QA:
- The Role : I prefer to be a QA then tester, here is a great article by Alan S Koch on Testing vs Quality Assurance. All the projects that I have worked my role was never limited to software testing. As QAs we are responsible for varied tasks ranging from software testing, writing automation code, creating test systems, data analysis, requirement analysis, release planning, risk mitigation, iteration planning, reporting, training and many more that varies with teams. We are the gatekeepers, our role is to assure quality of the whole product which is not limited to navigating through the application and writing code.
- The Work : As a QA my daily work involves both manually testing the functionality and writing automation code. I was once working with a business analyst who had recently moved from QA role, he confessed that he missed writing code. QAs understand the functionality and usability of the system, sometimes more then the business analysts. I have always been excited about exploratory and manual testing as much as automation aspect. I have seen people jumping to automation in the first week of their job, which always wonders me that how in the world can anyone do automation without understanding the application? These are the people who say that manual testing is boring but because of their lack of application knowledge are not able to get good code coverage.
- Quality : I have never been a bug hunter, not that I don’t find bugs. My aim of testing had never been finding bugs but to assure quality of the system. For me bug count has never been the metrics for evaluating the performance of testing team. Instead how early the testing team can provide feedback once the work is done is the true performance measurement. This approach works really well and dramatically reduces the number of bugs found close to release or in production. Earlier the issue is found and fixed lower is the cost incurred and better is the quality of the system.
- The Team : I have always enjoyed working in collaborative teams where I can talk to anyone at any point during my work. Creating protocols for communications reduces the work throughput. QAs always seek answers for their never ending questions to developers and product managers. Teams where QAs are supported well tend to perform better and are more efficient. But its the responsibility of testing team to make sure they are not bugging people with irrelevant question s which can affect the trust in the testing team in long run.
My experience as QA/tester had been amazing and it’s definitely a great role with lot of opportunities in the software industry. Its the responsibility of us who have been in this profession for long time to help and encourage new talent to learn the art of software testing.
BDD and Agile testing
BDD has gain popularity among many Agile shops in past few years. It has helped teams to work more collaboratively and with focus on end user expectations and business goals.
BDD with ubiquitous language gives the power to business and development teams to define the tests in business domain language. The Give…When…Then format let’s you write the tests in readable english.
Given I am a registered user
When I enter my credentials
Then I am on the user account page
Above scenario gives a very clear picture of the tests intent without diving into the underlying technology and tools used.
BDD framework gives the testing team an opportunity to provide their inputs to the story even before the development is kicked off. QAs can write BDD scenario before the story with Product team which can then be used for automation tests. This establishes a very cohesive model where acceptance criteria are translated into a well functioning feature drive by BDD scenarios. It proves to be very successful as the automation of the features is done right with the development and the QA cycle doesn’t lags behind the development cycle. BDD scenarios bridges the gap of documentation, lack of which is mostly feared by big organizations while adopting Agile. It covers for both requirements and manual tests which are difficult to maintain and often loose their worth in long running and big projects.
Given the wide acceptance of BDD in projects, frameworks are available for most of the popular technologies used, here are the few popular ones :
CAST 2012 - Day 2
Another day full of learning, fun and meeting amazing people. I have never been around so many smart people where every body is talking about testing. Talking to different people coming from different countries with completely different ways of working but the same goal - how as QAs we can help our organizations and people around us? These conversations have helped me clear my thoughts and re-affirm that the way I am working in my projects is the right approach.
Day 2 had more of workshops lined up then day 1, which makes it more difficult to chose which sessions to attend. I was introduced to Anna Royzman the day before as she also is based out of New York and she told about her workshop on Thought Provoking Leadership. I decided to attend her workshop for the first half of the day. In this workshop divided in different groups we used the Empathy maps from Game-Storming principle to understand the stake holder for a product and his needs as a stake holder.
We used the travel ads product that we have at IntentMedia to draw the empathy map in our group and we considered the customer who sees the ads as stake holder. I learned about our actual end user here more than I could in more than a year that I have been working at IntentMedia.
Post lunch and post the iPad draw(I didn’t win :( ) the keynote was delivered by Elisabeth Hendrickson. She talked about the theme of the conference ’Thinking Tester’ and I couldn’t have agreed to anyone else more than her on the topic that testing is dead. She started by explaining the testing in current perspective as 'Any activity that yields empirical evidence about the extent to which our intentions, our implementation and the actual business needs are aligned'. Most of the people think that testing is just checking but a code is tested when its checked and explored. Exploratory testing is a very important aspect of testing. Therefore a ‘Thinking Tester’ has following traits: analytical, relentlessly curious, observant, skeptical, empiricist, critical thinking, investigator. The next point that she mentioned was very true with how I think of my work and have been doing it for past 5 years. We in our jobs as tester are not only testers, we play different roles of Product Owner, Programmer, Architect, Project Manager and many more. She closed her talk which I believe explains how the testing community can continue to grow and play that important role in organizations: Testing is not Dead. But the Context continues to evolve, and so do we. Great close to one of the best talks I have ever attended, here are her slides for reference.
The second part of the day was again a workshop that helped me understand better my work as a service to various stake holders. Lynn Mckee held the workshop on topic Thinking about Testing as a service. During this workshops in our group we brainstormed about what we think Testing and Quality Assurance are. Often the terms that we used daily in our lives are often misunderstood and continue to be ambiguous. Next steps was to understand various stake holders for testing service in a project. These stake holders as we discussed usually are Programmers, Project Manager, Product Owner, Fellow QAs, Share Holders, Customer, Operations and many more. As QAs we directly or indirectly offer our services to each of these stake holders and this makes our job more crucial and important. Lynn has a great website which provides huge resources that would be helpful to any tester to improve his daily activities.
During CAST welcome note on the first day it was mentioned that in all other conferences evenings are very dull and everybody retards to their rooms but at CAST they have to ask people to go back to their rooms as it was late at night. Through tweets I found that even the first day there were people till 1:30 AM playing testing games. Therefore I decided to stay back the second day to find out more about these testing games and was there past 11 PM.
We played different games for more than 4 hrs, it was fun and yes there was so much learning from these games.