DevOps | Software Automation | Continuous Integration

Tag: Software Testing

Evolution Of Software Testing

I was lucky enough to be able to present in the Melbourne Selenium Meetup which was held at on the 11th of December, 2014. 
Below is the script of my speech with the title Evolution Of Software Testing.


Thanks to Ray for inviting me to the MeetUp. I am definitely not the most experienced here, just to share with everybody the changes in testing that I have observed in the past 10 years, which you may be experiencing too and my personal experiences in the journey.


I would like to share my experience in terms of these 4 areas.
(1) Transition from traditional to Agile
(2) Importance of automation 
(3) Battling between time and quality in a world when we need to deploy fast 
(4) What do we need to do to keep up in a rapid changing industry

Waterfall to Agile

Most of our changes started after the adoption of Agile SDLC over Waterfall

Testing is part of software engineering

Testing no longer is a phase in SDLC. It is part of software engineering. What does that mean?

Bug catching at every phase

Bug catching is not done at the final phase before deployment. Testing starts at early phase, during planning or design, before development starts.

Real Life Scenario

My experience with the transition from Traditional to Agile is that it helps to deliver better quality results and on time as it eliminates the back and forth bug catching and fixing which will cost more towards the end of a development phase. Testing bit by bit is gives a much better result than testing in a big chunk. This also helps to ensure that the product is what the stakeholder wants.
I heard about a real life scenario where Company X invested in a 4 year traditional software project. When the stakeholder 1st time sees it, it was a shock as it was not the product they expected.
During Waterfall, testing is the final phase before production and testers are always being pressured to test fast in order to deliver on time. My worst experience is to receive a project that would need a few days of testing at 3pm and needs to be deployed the next day. As a result of that, we have to work overtime, or take out features that have show stopper bugs. PMs tend to give a lot of pressure to the testers back in those days.

Get rid of the documentations

We no longer need to write piles of test cases or test plans.

Automation is the key

Automation is our documentation. Yes, we need to learn to code. Test cases or test plans are converted into automated tests.
No technology is perfect without a testing framework. Every new technology or language comes with a way to test it.
Web applications: Selenium
Mobile: Calabash, Appium, Robotium
.NET: Nunit, Xunit
Java: Junit
Angular JS: Protractor

Choosing the right tool

How to choose the right automation tool? These are the key criteria to consider. 
Most popular used
Large community support
Open source vs Commercial
Skills of the team
Personally I prefer Open Source tools as they are more fun to work on. 🙂 Or even custom built. For example, we use .NET library Web Client to test API and Web services

Real Life Scenario

So how do we start from doing manual testing to automation? Take baby steps. Start writing BDD scenarios instead of test cases. Then learn how to use Selenium to drive web browser. Then try more complex code manipulation before moving on to the framework level in making sure the test framework is reusable, scalable, maintainable. 
As you expand your Selenium usage, you’ll realize that we will face some challenges in certain areas such as pages with lots of AJAX calls. In that case we will need to have workaround these scenarios, like checking that all jQuery has been loaded.
Another Selenium problem is Element No Longer Attached To DOM. In this case, we will need to have a while loop and try to find the element with a catch for StaleElementException.

Test = User + Design

We need to use the product as a user and also understand the design to be able to think of the possible loop holes 
In the older days there is less emphasize on tester to be know the software design architecture. But in Agile, as we are involved since the earliest phase of SDLC, we will are expected to know to pick up not merely software bugs, but also requirements and design issues.
Knowing the software design helps to pick up edge cases. For example:
If we know that the database field has a size of 50 varchar, we can try to input >50 chars into it and see what happens. Will it crash or truncated?
If we know that a CSS file is used in multiple location, we will need to test multiple locations if it is changed

Time vs Quality

Nowadays we need to deliver software fast. Deliver, get feedback, or fail fast is the best practice. So how can we do that without sacrificing quality? I used to fight with our PM back in he old days in order to get the best quality product out the door. But nowadays that does not work anymore because time = money for business and as a QA, we need to work towards the same goal too (management will like this).
We can deliver software with known bugs. We do this by understanding the critical path of the system usage and ensure that most red route are working correctly, while some bugs can be left in some feature that has minimal usage. We can use data or user analytics to understand which are the most visited pages, most used features, etc to help us make the decision.
My personal experience is when I started at my current job where we do not automation at all and testing happens after deployment. Our product is complex and I am having time pressure to put some automation in place.
How we go about to solve this problem is to find the top 20 pages which equals to 90% of usage. We then analyze the data and pages and put them into a logical user flow. With that we then translate that into BDD language and automate the scenario based on it.

How To Stay Alive

So software testing has changed so much and it is constantly changing just like everything is. How can we keep up with this?
We need to constantly learn. Don’t worry if there isn’t enough training budget (management will like me once more), as learning doesn’t always cost $.
There are many different channels that we can learn which are free. These are some of my personal likes. 
My personal experience with learning are: Examples:
New tool: Appium by Jonathan Lipps at GTAC 2013
New ideas: Take flaky tests from CI. Put is back when they are stable. Critical tests should never be taken out by Ankit Mehta at GTAC 2014
New ideas: Turn off a feature that is buggy by Ankit Mehta at GTAC 2014
I then I like to Blog about what I have learnt. Its not just serves to share with people what I’ve learned, but also to keep a note on what I’ve learnt.
We also need to get our hands dirty and get some hands on experience with what we learned. We can only really learn something when we work on it and gain experience from it. When I face problems I will post questions onto forums like StackOverflow, or Google forums. 
Lastly, share and give back to the community what we learnt. Try to help people out there that are learning just like us.
And most important of all is to love what do! As what Steve Jobs says.Thank you.

Future Of Software Testing

For the past few years there have been a lot of demand for automated testers. Companies are replacing manual regression test checklists with automated scripts, especially the ones written with Open Source tools such as Selenium.

However, I started to see job advertisements with graduate automation testers and developers with knowledge of testing tools.
Therefore, as time goes, software testing job or even automated testing job might be fading away as well, just like manual testers today.
So where do I see the future for software testing? I will see software testing as part of software development. Software testers will be focusing on the impact of the changes to other areas of the systems or edge cases finding. They will find them and fix them as well. They will also be focusing on performance and security aspects. They will also step in as a business analyst, product owner, or even iteration manager when needed.
Basically, what I see the future of software testing is to be a multi-skilled member of a software development team. I urge software testers to start improve their technical, communication, and business skills. Performance testing and security testing knowledge will also be considered as a basic skill of software tester instead of a specialist’s skill.
In this modern day, especially in IT where everything is changing dramatically, changes in a process or role is unavoidable. Therefore, keep learning and keep improving!

What Does A QA Do?

I held a learning session with my team a couple of weeks ago about this topic “What Does A QA Do?” The reason is because I think as we are so much driven into the daily tasks and responsibilities to deliver projects and hitting deadlines, sometimes its good to take a pause and rethink, “What are you actually supposed to be doing?”
Who are we?
There are many definitions of the role. QA, tester, test analyst, test engineer. Regardless of what our official title is, I think we are simply part of the development team that help the team to deliver a quality and successful product.
What do we do?
Most would say we find bugs and raise them. Correct, but not just that. We link business requirements with technical design, into a quality end user product.
How do we do our job?
There are many technical terms for this, manual, exploratory, automation, regression, performance, security, etc. Firstly, we need to have the knowledge of a product manager to know what product are we building. Secondly, we need to have the knowledge of a business analyst to understand the detailed business rules behind each piece of functionality built. Thirdly, we need to have the knowledge of a developer to understand the system design,  and technical solution such as what does the code do, the data flow into the database, etc. Lastly, we need to be able to use the product as an experienced end user.
What type of bugs we find?
Most would think that bugs are software bugs, careless mistakes by the developers. This is only a minor part of it. Bugs can be:


  • Product bug – building the wrong product from the very beginning, wrong idea
  • Requirement bug – caused by lack of understanding of the business rules and outside the square thinking
  • Software bug – caused by wrong implementation
  • Environment bug – caused by the way the software interacts with an operating system, system configuration, database version, etc
  • Deployment bug – caused by sequence of deploying (e.g.: running a script prior update of code might break a system), code merge mistake, database backward compatibility issues
  • Performance bug – all the above bugs are eliminated, but we forgot to take into account that the system might break down under peak usage
  • Security bug – all the above bugs are eliminated, but we forgot that there are back doors or windows opened that allow intruders in to break the system and do nasty stuff!
I am not trying to over complicate things. Many would think QA only test. Yes, but a good QA does not only test. A good QA should have the following skill sets:
  • Good understanding of business knowledge
  • Able to think outside the square
  • Good communication skill with technical and non-technical stakeholders
  • Good team player
  • Good technical skills



Principles Of Regression Test

I guess I have posted a lot of technical stuff in my blog. However, I would like to go into a more theoritical side of things about software testing today – how to write a regression test.
So, what is regression test? Everyone knows that it is a set of test to make sure that existing functionality still works after new functionalities are added.
But how to write a good regression test? Most would think that the more tests you add into a regression test suite the better it is. This is a wrong view about regression test.
There are rules to follow in order to write an effective and efficient regression test suite. These rules are:


  • Test all the core functions of the system (not the little help screen)
  • Test the happy path and user would follow (not edge cases)
  • Partition it logically based on system or core modules so that it is easily readable and accessible
  • Partition a set of tests that will be run before every release (very important core modules) and others that will only be run when necessary (changes occur)
  • Keep it as simple and short as possible
  • Avoid repetitive steps
However, please bear in mind that I am not trying to say we do not want to test more in regression test. I simply mean that most edge cases and low level tests should be in the unit test level instead of functional regression test.

Software Testing New Year Resolution 2013


  • Continue testing
  • Testing beyond testing (looking at business requirements, technical details, process, environment, etc as a whole)
  • Continue to read blogs/web about latest trend, tools, and technologies in software testing
  • Continue to identify tests that can be automated
  • Continue to explore and experiment the tools and technologies for test automation
  • Be creative and innovative in building test automation
  • Continue to write about what you have learnt in your blog
  • Go beyond a tester’s role (help identify cause of bugs and fix it if you can)
  • Enjoy what you do 🙂

The Art of Test Automation

There is no one or absolute way for building test automation. I use the following approach to choose the technology or tool I use to build my test automation framework:

  • Open source
  • Most popular in the community
  • Regular maintenance by community
  • Able to solve my problem
Below is an example of the approach or tools I use to write a regression test suite for a customised exporter script in my company.
The behaviour of the exporter script is as below:
  • Log into a web interface
  • Fill in details as SQL statement to be run into a web form
  • Kicking off the exporter script via web form
  • Exporter script will execute the SQL statement and generate the output as text file
  • The text file will be sent to a FTP account
Below is how I build the test automation suite:
  • SpecFlow

I use this BDD tool for user readability purpose to explain the flow of the test scenarios

  • Selenium web driver in .NET

This is used to do the web interface interactions such as log in, filling in SQL details, and kicking off the tests

  • WinSCP

To log into the FTP server

  • Batch script

To run WinSCP.exe in order to delete and download files from FTP server

  • C# code

Using the Process class to run the batch script. It is also used to verify and display the output.

  • NUnit

To verify output

Below is the example of going down into each level of 1 step in the scenario:
The SpecFlow scenario looks like this:
Scenario: Kicking off PageUp Exporter
    Given I delete the output file
When I log into Feature Manager
And I kick off an Exporter process with query “SELECT * from dbo.applicant where semail like ‘%test%’ and linstid = ‘543’”
Then I verify that the file is exported
And we verify that the file has “60” rows
And we verify that the file has “100” delimiters
And we verify that the email address contains “test”
And we verify that the file size is “19484”

The step definitions for the 1st step consists of C# code that calls the batch file:

 [Given(@”I delete the output file”)]
        public void GivenIDeleteTheOutputFile()
            string fileName = @”c:PageUpOutexporterTest..txt”;
            Process p = new Process();
            p.StartInfo.UseShellExecute = false;
            p.StartInfo.RedirectStandardOutput = true;
            p.StartInfo.FileName = “C:\exporter_delete.bat”;

The batch file calls a text file which WinSCP needs:

cd c:Program Files (x86)WinSCP
start WinSCP.exe /console /script=c:exporter_delete.txt

The text file consists of more MS DOS commands to perform the operation on the FTP server:

# Automatically abort script on errors
option batch abort
# Disable overwrite confirmations that conflict with the previous
option confirm off
# Connect using a password
# open s -hostkey=”ssh-rsa 1024 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx”
# Connect
open s -hostkey=”ssh-dss 1024 9e:bd:6f:c8:f9:68:18:81:f0:50:ce:71:dd:d6:53:2e”
cd /
# Download file to the local directory d:
rm *.txt
# Disconnect
# Exit WinSCP

© 2023 Chuan Chuan Law

Theme by Anders NorenUp ↑