Experience on the Application of Distributed Testing in an Agile Software Development Environment

Software engineering is by nature a highly collaborative activity. However, this collaboration is more difficult when the teams are geographically separated, as several factors, such as work-time, cultural differences, communication, technical capability, among others, may impact on its success. Moreover, each activity in the software development process has specific needs in a distributed software development (DSD) environment.

In this post, I will report an industrial experience of a testing team separated geographically in the context of a software project that followed an agile method. This experience was possible through a collaboration between industry (Nokia Institute) and the federal university of amazonas, and its generated a paper presented in ICGSE 2012.Authors: Eliane Collins, Gisele Macedo, Nayane Maia and Dr. Arilo Dias-Neto.

APPLYING AGILE TESTING IN A DSD ENVIRONMENT

A. Software Project Characteristics

This project had been conducted in the context of a research institute in Manaus/Amazonas/Brazil aiming at

developing a web application to manager advertising campaigns, called Project X. This software project was developed following the Scrum agile methodology , and it consisted of 30 stories (Product Backlog) divided into 9 Sprints (iterations) varying from 2 to 3 weeks. This project had a development team composed of 1

Scrum Master and 3 full-time developers. The Testing Team, the focus of this paper, consisted of 6 professionals, being 2 full-time testers located in the same physical environment of the developers and four part-time testers who worked in another site (geographically separated) as a result of a cooperation between the research institute and the Federal University of Amazonas (UFAM).

The Project X was developed using a web platform (PHP + MySQL database), and IDE Eclipse. The system was composed of 8 screens (4 forms to register campaigns and users and 4 screens to search and generate reports).

B. Testing Process

The Project X’s software testing process followed the Scrum methodology, where the testing team was integrated to the project team, participating in Scrum ceremonies (sprint review, daily, retrospective and planning meetings). Their responsibilities were to plan test cases through the stories described in the Product Backlog, specify the acceptance criteria, and use test automation tools to speed up the execution activities of each sprint.

Due to the geographic distribution of the testing team, the testing process was designed considering the characteristics of a DSD (distributed software development) environment. We needed to provide a testing structure allowing access to the project’s information by all team members. Thus, in order to attend this demand, the testing team used a server dedicated to be connected remotely with the following tools:

• TestLink: used to manage test plans, write test cases, and report tests execution. Besides, the selection of tests to compose a test suite was done manually, TestLink acted as an editor and organizer of test cases, storing all information. It facilitated the creation of test plans and reports documentation, and the controlling of the tests execution versions;

• Mantis Bug Tracker: through this tool, which was already used in the institute, the tester registered the

defects found, sent them to the developers and controlled the lifecycle of each defect. 

• FireScrum: a web tool for Scrum taskboard was used to detail tasks defined for the project sprint. This tool was used for both testing teams (remote and local) to record and update the tasks progress. Thus, the test coordinator used these as data in the project daily meeting.

• Subversion: it was used to share and manage information among the test team members. 

Moreover, in order to ensure the operation of this infrastructure and facilitate communication among the teams, a Test Leader located in the institute was responsible for defining the testing tasks, planning the sprint activities and reporting the progress of these activities at the daily meetings. Only the Test Leader could communicate with the development team, avoiding a larger communication network in the testing team. The remote testing team was allocated to the tasks of designing, automated (creating scripts), and executing test cases for the stories developed in each sprint.

The tasks of test process during the sprint were incremental and iterative following the scrum process. They are represented in Figure 1. The dark rectangles represent the Test Leader’s activities and the lighter rectangles represent the Remote Testing team’s activities.

Figure 1. Overview of  Testing Process

The task Incremental Test Execution includes execution of exploratory tests and automation of regression tests. For Project X project, the tool chosen for the functional tests automation was the Selenium RC and IDE5. Selenium uses the approach record-play support and tests web applications for the browser Mozilla Firefox. Selenium IDE can record the user actions in the browser, creating test scripts in several programming languages and executing them later. The Selenium Java API used in the project allows running the test scripts in other browsers such as the versions of Internet Explorer. The automated test suite is updated and executed in every sprint. The tools TestLink and Selenium were integrated, thus, when

test cases are executed in the Selenium tool, their results are recorded automatically in TestLink.

C. Communication Process

According to agile practices, communication is an essential factor for the success of the agile project. Project members who have good communication process can cooperate more and mitigate risks of changes during the project. It is an essential factor when some teams of the project are geographically distributed.

To work with distributed teams, it is important to use tools to facilitate communication. In this project we used several tools like e-mail, a free scrum tool for task board (FireScrum), and the chat and video-conference tool (Skype) to make possible online communication among team’s members, resolving doubts as soon as possible. Thus, e-mail was used only for offline communication (invitations to meetings and

project documents). 

Daily, all team members needed to access the online FireScrum tool, where the tasks were created and allocated to the testing team members by the Test Leader. The Remote Testing team was responsible for updating the FireScrum tasks every day, reporting the progress and impediments.

Figure 2 represents the communication flow among the project teams. According to the flow, communication between the Development Team and the Test Leader was straight (face- to- face) while communication with the Remote Testing team occurred through the server test tools.

Figure 2. Communication Network.

D. Tasks Allocation

Using the Scrum methodology, tasks are defined and estimated in Sprint Planning Meeting. In this ceremony, each professional involved must participate to specify the tasks necessary to accomplish the stories. All members were invited to this event, including the remote team. On presentation of the definition and clearly understanding the tasks, the Test Leader inserted them into the FireScrum tool and allocated them to the testing team members. During the sprint, the Remote Testing team could add and assign new tasks, when appropriate. With all communication tools available, any possible impediment from the remote testing team could have been quickly detected and easily solved avoiding delays. Each professional team was responsible for his/her external tasks and he/she had to communicate any eventual difficulty as soon as possible. The task status could be defined as: to do (to be made), in progress (ongoing task) and done (tasks completed).

Every day, before the daily meeting of the project in the company, the Test Leader checked the progress of tasks to inform the development team. Among the main tasks assigned to the Remote Testing team, we can cite: specification of new test cases, update of test cases from previous stories, creation and automation of

test scripts, execution of test cases and test scripts of the stories in sprint, defects registration, execution of automated scripts to regression tests and validation of solved defects.

The main Test Leader’s tasks can be summarized in: review created test cases, monitor tasks in FireScrum, facilitate test tasks, send project information and report test execution.

E. Planning and Restrospective Meetings

Following Scrum, there are two other important meetings to be performed: planning and retrospective. Planning is the meeting where the whole team decides to express their opinion about how to get the stories according to the prioritized backlog. In Project X, all testing team participated in this meeting (including the remote team), choosing the stories to be developed and estimating the complexity of the selected stories. The retrospective meeting is considered a very important ceremony because it is where is said what went right and what should be improved for the next sprint. All team members, including the remote testing team, participated in this event, sharing experiences and understanding the difficulties that would be improved in the next sprint. With the participation of the remote testing team in both Scrum ceremonies, it’s possible to get the unit of the project team and everybody feels as part of the team.

CHALLENGES AND LESSONS LEARNED

In this experience on conducting distributed testing in an agile software project, we could observe it can work very well. However, some issues need to be managed to avoid the risks introduced by the combination of these software engineering practices (DSD + agile practices). Thus, we identified some challenges and key lessons that we learned for minimizing the impact of the geographical distance between the testing teams:

A. Communication and Coordination are essential factors for the success of distributed testing All main research works in the DSD field indicate communication and coordination as important aspects to carry the success in a distributed software project, and we could also confirm these claims. Some practices were essential to reach this success:

• Allocating one person (test leader) as a link between the local and remote testing team is very important to avoid a large communication network and, consequently noises. Thus, all information and decisions would always pass through this professional, responsible for distributing information, solving impedances and communication problems, and making the testing tasks easier.

• A communication protocol should be formalized, regulating how the teams should keep contact with each

other. That includes structure of emails, communication tools, meetings, timetable, and so on. This is essential to avoid losing data, effort, and quality in the software project. For instance, the definition of a standard for test cases specification and bug reporting allows a tester to complete a task started by another tester or validate a bug reported by another professional.

• An online chat tool should be used and the key persons in the project (Scrum master, development and test leaders) should be always available to clarify doubts.

• Periodic meetings between the development and testing teams should be scheduled. This contributes to understand the complex user’s stories. Moreover, the physical presence of the remote testing team in the

Scrum meetings (Planning, Retrospective and Review) is very important to share, among all software project

members, information regarding problems, suggestion of improvements and planning of new activities.

B. The project information should be available with details to all members As part of the testing team is separated geographically, only short stories and acceptance criteria descriptions are not enough for test cases’ specification. Thus, user’s scenarios and the application wireframe should be available to cover all features to be developed/tested, bypassing the limitations imposed by the distance between part of the testers and the product owner. Moreover, changes in the user’s scenarios result in a high effort to update test cases and their automation scripts. Therefore, these scenarios should be kept updated, otherwise they can affect the quality of the testing activity.

C. Automation reduces the needs of physical presence in the testing process The tests automation has an essential role in the success of distributed testing, because both teams would be able to run the regression tests suite constantly without depending on time zones between the remote and local testing teams. Moreover, automation speeds up the time required for running the tests even when new releases are published by the development team close to the deployment deadline.

D. Supporting tools are always important, but testing team organization is more Tools are also highlighted in the technical literature as an important element in a distributed software project. In the context of distributed testing in an agile project, two special tools should be cited:

• A Scrum dashboard tool presents all activities performed daily by the remote testing team, showing information regarding the performance of testers and the tasks’ progress. Thus, the test leader is provided with information to support the testing process management, mitigating eventual risks. In our software project we used the FireScrum tool.

• A test management tool controls all information regarding the testing activities, test cases specification and validation, creation of new releases, test running, number of detected failures, and bug tracking (creation, correction, and validation). In our software project we used the TestLink and Mantis tools. Besides their importance, tools can fail, and we need to be prepared for this moment. If the network or a server crashes, all remote teams’ activities can be affected because communication between the teams will cease to exist. Thus, we cannot be dependent on tools to perform our activities. 

CONCLUSIONS

We could observe that it is feasible to integrate the DSD with the agile practices, maintaining management and organization of a software project, because some of them are considered in the testing activities for both scenarios, as the need of an efficient communication, automation as a resource to reduce cost, and task allocation in small parts. On the other hand, the difference in these scenarios regarding the testing activities, such as continuous integration and daily meetings, could be avoided with some technological solutions already reported in the technical literature.

Some challenges and lessons learned could be extracted from these experience and they could support other software engineers when performing testing in a similar environment.

Complete paper: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6337365&isnumber=6337293

Using Fitnesse Test Tool for Acceptance Test in Mobile Applications

There are few tools to make acceptance tests for mobile applications, eg J2ME platform.

The tool J2MEUnit seems a good solution for unit testing tools but compared to Fit frameworks, the tests become more cumbersome. 

So I decided to use the test tool Fitnesse to validate business rules in a J2ME application and the tests implemented were simpler and faster. See the instructions  to do Test-First approach.

Configuration

To use the tool Fitnesse into eclipse, it is necessary to install Fitnesse plugin for eclipse.

1) Start up Eclipse and select the HELP menu -> Install new software

2) Click the Add button and type in the Name field: Fitnesse and field Location: http://www.bandxi.com/fitnesse/ or usu this Link.

3) Select Ok

4) Select Fitnesse Plugin for Eclipse and then click Next.

5) The Eclipse Plugin will be installed and will restart.

6) When eclipse restarts, the two icons of Fitnesse appear on the toolbar.

Create a Project

1) In eclipse, you click the File menu and select New-> Project.

2) You can select a JavaME Middlet Project.

3) Give a name, for example: FitnesseProject.

4) Select your new  project in the left of the eclipse package explorer tab.

5) Select the project created with right click and select Fitnesse -> Add to project classpath libraries fitnesse. 

6) Then click again with the right click over the project and select File -> New -> Folder to create a folder.

7) Give the name of the folder: fitnesse

8) Right-click the folder and select Fitnesse-fitnesse> Lunch Fitnesse runtime on this folder.

9) In the name of the project click the right click and select Configure Buildpath

10) In the window to configure Buildpath, add the following jars: 

• collections.jar
• ee.minimum.jar

It is necessary because of fitnesse dependences.

11) In your test project, select the src folder with the right click and select: New -> package, give the name of the fixtures this new package.

12) Build a fixture to test for example an calc method.

13) In your test project FitnesseProject go to fixtures package and right click and choose New-> Class

14) In the window to create the new class type the name CalcFixture and in superclass field, click Browse and enter in the window ColumnFixture.

15) Select the class from the list and select Ok to create the class.

16) Write a test fixture with the following code:

package fixtures;

import fit.ColumnFixture;

public class CalcFixture extends ColumnFixture {

public int a, b;

  

      

    public  int result() {

   

        return calc.add(a, b);

    }

}

The line  return calc.add(a, b); is gonna be red because the method add does not exist.

Lets create the system code:

17) In your project, create a new package called app (src right click and select: New -> package,).

18) Create a new Class in this package called Calc.

19) Insert a code just to fix the error in your fixture.

package app;

public class Calc {

public int add(int a, int b){

return 0;

}

}

Now the error in your fixture will disappear.

Executing Tests 

1) In the eclipse toolbar, click the icon to launch the Fitnesse server.

2) the eclipse browser will appear in the eclipse fitnesse page.

3) the page is similar to a Fitnesse wiki, click the Edit menu on the left top of the page.

4) Enter the word anywhere ProjectTest and save the change.

5) You will see the word followed by a question ProjectTest?

6) Click the interrogation ? mark.

7) You will see an edit page of a wiki.

8) Enter the following code in the wiki:

!path /C :/workspace/FitnesseProject/bin

! 3 Column Fixture

|!-fixtures.CalcFixture-! |

| a | b | result () |

| 1 | 2 | 3 |

| 13 | 5 | 18|

It is important to point to the bin folder of the the project using !path.

To indicate the package and fixture class to be used, we use the command |!-packagename.classname-! |

Then the decision table is implemented with the parameters to be passed, text separator and the expected return result class ().

9) Save the page in the wiki.

10) In the page saved, select the blue menu on the left Properties.

11) On page properties select Test and then click Save Properties.

12) When you return to your test wiki the button Test will appear in the menu.

13) Click the Test to run your test.

14) If a message in yellow saying there was not found the test class or fields, make sure the project generated classes in the bin folder or you are pointing to the correct address in the bin folder.

15) Another important check is if you put the exact same name as the parameter of the fixture in the table.

16) If the test run the errors in red will appear as the picture bellow:

The errors appear because the method add in the class Calc wasn't implemented.

17) Back to project and open the class: src -> app -> Calc.java.

18)Insert the code:

package app;

public class Calc {

public int add(int a, int b){

return a+b;

}

}

19) Save and back to Fitnesse page in the eclipse.

20) Don't forget to build the project.

21) Execute the fitnesse tests again clicking in Test.

22) The test table will be green, your test is passed as the picture bellow.

23) When your class will be totally tested, you can create a middlet and call your class Calc.

24) In src, right click and create the package with the name mobile in New-> package.

25) In the package created click the right mouse button and create a midlet. New-> midlet

26) Put the name of the HelloWorld midlet for example.

27) In midlet created add a private variable private Display display;.

28) In the startApp () put the following code:

Display display = Display.getDisplay (this);
String message = String.valueOf(new Calc().add(3, 4));
TextBox textBox = new TextBox ("Hello World", message, 50, 0);
display.setCurrent (textBox);

30) Run the midlet selecting the project and clicking the right click and selecting: Run As -> Emulated Java ME Midlet

31) The midlet will run Hello World and will appear the result of  add:  7.

For the next functionalities, repeat all the process :)

Fitnesse pros:

- A fixture class tests one class of the system.

- Parameters are passed only in wiki and it is not necessary one test method for each parameter (as JUnit).

- It is friendly and developers and Testers can use the eclipse itself.

Fitnesse cons:

- Tests are run only in eclipse and it is not possible to embed the tests in the emulator or in the cell phone as J2MEUnit do.

My Conclusion is:

If you want to validate business rules and acceptance criteria, the Fitnesse works well and fast. But if the developer wants to analyze the memory limits and the cell phone performance, so J2MEUnit and Manual tests are required. 

Good Links:

• http://fitnesse.org/

• http://schuchert.wikispaces.com/FitNesse.Tutorials.0.Java#Create Eclipse Project

• http://softeng.polito.it/courses/tutorial/FitnesseInEclipse.html

Starting to use Selenium Webdriver

For a long time I used the Selenium IDE and RC for web system testing. Using java platform, it was possible to build robust test suites and generate automatic reports using Junit.

Like last year I have been worked in systems for mobile devices, I had not yet experienced the successor of Selenium RC, the Selenium Webdriver.

Selenium webdriver is a tool designed to provide a simple interface and a concise programming to better support dynamics web pages.

In my research about Selenium Webdriver, I found a interesting comparison table that I show bellow. It can help to understand the differences between Selenium RC and Webdriver.

Selenium IDE	Selenium RC	Webdriver
Works only on mozilla	Works on almost all browsers.Does not work on latest version of firefox/IE	Works on latest versions of almost all browsers - Firefox, IE(6,7,8), OPera, Chrome
Record and run tool	No Record and run	No Record and run
No server required to start	Server is required to start	No server required to start
Core engine is Javascript based	Core engine is Javascript based	Interacts natively with browser application
Very simple to use. If using User extentoins, you require knowledge on javascript which makes the work a little bit tough.	Its a simple and small API	Complex and a bit large API as compared to RC
Not at all object oriented	Less Object orinted API	Purely Object oriented API
Cannot move mouse with it	Cannot move mouse with it	Can move mouse cursor
Full xpaths have to be appended with 'xapth=\\' syntax	Full xpaths have to be appended with 'xapth=\\' syntax	No need to append 'xpath=\\'
No Listeners	No Listeners	Implementation of Listeners is provided
Cannot test iphone/Android applications	Cannot test iphone/Android applications	Can test iphone/Android applications

source: http://qtpselenium.com/selenium-tutorial/difference-between-ide-rc-webdriver/

I found very interesting the following points:

1. In Selenium Webdriver is not necessary to initialize the RC server.
2. It is not necessary to use XPath.
3. Fully object oriented.
4. You can move the cursor.
5. Can test on iphone and android applicat

For these reasons I decided to try the Selenium Webdriver.

I will describe my first example:

Installation and Configuration

1. In Selenium page http://code.google.com/p/selenium/downloads/list  download the following  files: selenium-server-standalone.jar, selenium-java-2.29.0.zip , and the browser drivers : IEDriverServer_x64_2.29.0.zip and chromedriver_win_23.0.1240.0.zip .

1. Open the Eclipse newest java version.
2. Create a Java Project . File -> New -> Java Project
3. Choose a  name for your  project,  SeleniumTest for example.
4. The new project will appears in the package explorer.
5. Select the project created in the package explorer and click using right mouse button.
6. Create a new folder : New -> Folder
7. The name of new folder is lib.
8. Extract the selenium-java-2.29.0.zip and copy selenium-server-standalone.jar to inside the new folder lib.
9. Select the jars and click using right mouse button and select Buildpath -> Add to Build Path.
10. The IEDriver and Chromerdriver must be stored in a folder in the C:\BrowserDrivers.

Fast Example

Let's try to test the login page of testlink demo.

1. Open the firefox and Install the Selenium IDE (http://seleniumhq.org/download/). 
2. After Install the Selenium IDE, start it in Firefox -> Web Developer -> Selenium IDE.
3. The Selenium IDE will open and start to record the actions in the browser.
4. In firefox start open the page  of testlink demo  http://demo.testlink.org/tl194demo/login.php.
5. In the Login Name enter: admin 
6. Password enter: admin
7. The testlink home page will appears.
8. Stop the Selenium IDE record.
9. In the Selenium IDE window, save the test in File-> Export As -> Java /JUnit 4/ webdriver.
10. save the file with the name Login.java in a folder.
11. Go to Eclipse IDE
12. Select the src folder in your java project created .
13. Create a class using right click and select New -> Class
14. The name of the Class is: TestLogin
15. Open your test recorded Login.java, copy the content and paste in the eclipse TestLogin class. Bellow the class example.

package example;

import java.util.regex.Pattern;

import java.util.concurrent.TimeUnit;

import org.junit.*;

import static org.junit.Assert.*;

import static org.hamcrest.CoreMatchers.*;

import org.openqa.selenium.*;

import org.openqa.selenium.firefox.FirefoxDriver;

import org.openqa.selenium.support.ui.Select;

public class TestLogin {

  private WebDriver driver;

  private String baseUrl;

  private boolean acceptNextAlert = true;

  private StringBuffer verificationErrors = new StringBuffer();

  @Before

  public void setUp() throws Exception {

    driver = new FirefoxDriver();

    baseUrl = "http://demo.testlink.org/";

    driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

  }

  @Test

  public void testEWebdriver() throws Exception {

    driver.get(baseUrl + "/tl194demo/login.php");

    driver.findElement(By.id("login")).clear();

    driver.findElement(By.id("login")).sendKeys("admin");

    driver.findElement(By.name("tl_password")).clear();

    driver.findElement(By.name("tl_password")).sendKeys("admin");

    driver.findElement(By.name("login_submit")).click();

  }

  @After

  public void tearDown() throws Exception {

    driver.quit();

    String verificationErrorString = verificationErrors.toString();

    if (!"".equals(verificationErrorString)) {

      fail(verificationErrorString);

    }

  }

  private boolean isElementPresent(By by) {

    try {

      driver.findElement(by);

      return true;

    } catch (NoSuchElementException e) {

      return false;

    }

  }

  private String closeAlertAndGetItsText() {

    try {

      Alert alert = driver.switchTo().alert();

      if (acceptNextAlert) {

        alert.accept();

      } else {

        alert.dismiss();

      }

      return alert.getText();

    } finally {

      acceptNextAlert = true;

    }

  }

}

1. Fix the package name errors and the name of the class.
2. In eclipse package explorer select TestLogin.java with mouse right click and Run As -> JUnit
3. The Login recorded will run.

Adjusts

One of the resources of Selenium webdriver is run the same test for more than one web browsers. Some adjusts in our first example test is required.

1. First create more two drivers variables :  private WebDriver driver1, driver2, driver3;
2. In the setUp() method replace for the following code:

   driver1 = new FirefoxDriver();

  driver1.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

File file1 = new File("C:/BrowserDrivers/IEDriverServer.exe");

System.setProperty("webdriver.ie.driver", file1.getAbsolutePath());

driver2 = new InternetExplorerDriver();

driver2.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

File file2 = new File("C:/BrowserDrivers/chromedriver.exe");

System.setProperty("webdriver.chrome.driver", file2.getAbsolutePath());

driver3 = new ChromeDriver();

driver3.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

                

  baseUrl = "http://demo.testlink.org/";

To run my test was necessary to include the browser driver address in the computer (File file1 = new File("C:/BrowserDrivers/IEDriverServer.exe");) to start the IE browser and Chrome browser.

Change your test code adding the tests for each browser. See bellow:

 @Test

 public void testLoginFirefox() throws Exception {

 

 driver1.get(baseUrl + "/tl194demo/login.php");

     login(driver1);

 }

 

 @Test

 public void testLoginIE() throws Exception {

 

 driver2.get(baseUrl + "/tl194demo/login.php");

 login(driver2);

 }

 

 @Test

 public void testLoginChrome() throws Exception {

 

 driver3.get(baseUrl + "/tl194demo/login.php");

 login(driver3);

 }

 public void login(WebDriver driver){

 

 driver.findElement(By.id("login")).clear();

 driver.findElement(By.id("login")).sendKeys("admin");

 driver.findElement(By.name("tl_password")).clear();

 driver.findElement(By.name("tl_password")).sendKeys("admin");

 driver.findElement(By.name("login_submit")).click();

}

@After

 public void tearDown() throws Exception {

   driver1.quit();

   driver2.quit();

   driver3.quit();

   String verificationErrorString = verificationErrors.toString();

   if (!"".equals(verificationErrorString)) {

     fail(verificationErrorString);

   }

 }

All drivers must be closed using driver.quit() function.

Save the changes and Run the TestLogin.java again.

Run As -> JUnit.

As result the Login test will be executed for 3 browsers: Firefox, IE and Chrome browsers.

There's no need to start the Selenium server before the tests and without effort it was possible to run tests for  different kind of browsers.

I will try to test using resources for mouse cursor moving and I will post here the results.

Bye.

How I Automate Tests...

To automate tests, first I think the requirements that I need to start.

The figure 1 bellow shows the inputs and the outputs that I consider in test automation.

Figure 1 Test Automation Inputs and Outputs

To start the test automation activities I need the inputs:

1. Test Cases selected to be automated, they must be repeatable.
2. Test Data.
3. Test Tools environment configured and preferably Continuous Integration.
4. Code, even using TDD (test driven development) the code (can be fake)  is needed to run the tests.

As desirable outputs of test automation, basically I consider the following artifacts:

1. Test Scripts.
2. Execution Log.
3. Results Report.
4. Traceability (tests and requirements or tests and failures) and code coverage.

Depending of  test tool, other metrics could be extracted from test automation.

During the project a lot of external events can happen to disrupt the activities of automation as changes in requirements, extra requests from customer, change of deliverable dates and etc. For this reason I adopt some procedures listed bellow to automate tests and execute the software test activities.

1.  Plan the test cases for the priority functionalities developed using a test tool (for example Testlink). 
2. Select the test cases to be automated (selection can be according to acceptance criteria).
3.  Automated test scripts creation according to test cases selected to be automated. These test scripts must be possible to be edited and gathered in a test suite, and reused to others test cases;
4. Update the test scripts is important to avoid wrong results.
5. Then, the testers executed the automated tests using a test tool and the exploratory manual tests also. The defects found were registered in Testlink and in the defect tracking tool.
6. The automatic regression tests for all functionalities  must be executed also when defects were validated to ensuring that another part of system was not affected by code changes.
7. The automated TestLink reports can be generated. The development team released another system version and the same activities process began again. 

I call these procedures Test Iteration because it will be repeated again for next version of software until the end of project.

I executed it in Scrum projects and Waterfall projects as well.My experience is reported in my paper "Iterative Software Testing Process for Scrum and Waterfall Projects using Open Source Testing Tools Experience". It was presented in the event ICTSS 2010 (International Conference of Testing Software and Systems).

Executing these procedures is possible to ensure the test automation activities in the project and avoid problems as outdated test scripts.

According to project platform is possible to use many test automation tools, for example:

Web projects: Selenium Test tool and Jmeter

For desktop projects (Java AWT or Swing) : Marathon test tool

For C# or Java application is possible to use Fitnesse test tool

For mobile J2ME applications : J2MEUnit

My favorite tools to use are Selenium, Fitnesse and Jmeter. In next posts I will detail how to use them.

Software Test Automation Means...

I use the definition of the book Automated Software Testing [1], it says that Software Test Automation means to automate software testing activities including the development and execution of test scripts, verification of testing requirements, and the use of automated testing tools. Automating the test activities can reduce the effort of manual and repetitive tests and can provide quick results feedback for project team.

I consider that Test Automation is part of Software Development (not just part of Software Testing). It is a multidisciplinary activity since it envolves programming skills  and testing skills, furthermore, the collaboration between the project team is a key factor for the success of this activity but that relationship I'll explain in another post.

According to Mark Fewster and Dorothy Graham [2] Automating tests is a skill very different from Testing, because knowledge of good tools and software architecture is required. But in order to gain benefits the project team must wonder why to automate tests, what automate, when to automate testing and the risks of automation.

Why to Automate Tests?

First , we all need to be quick (time to market).

In second, commonly in a system there is a set of many test cases to be run several times and soon, when they are executed only manually is risky because all tests can not be performed very carefully and effective all times due to factors such as the short time to delivery results, tiredness and even human distraction. In this sense Test Automation provides the execution of many repetitive tests quickly and facilitate the generation of documents. So, the test professional can dedicate time to execute tests that agregate value to the project.

Other reason to automate tests is the cost. The initial investiment is high, the team need to be trained, need time to analyze tools, buy tools and there is the learning curve to get test automation effectively in the project. But when the tests are automated and organized, they cost is cheaper than running manually because what needed to be run on a day for more than a person can run automatically at one hour and the results are generated fast.

Some other benefits for the project include: improvements on software development lifecycle, software reliability, quickly regression tests, generation of documents, failures monitoring and reproducing defects.

What Automate?

The candidate tests to be automate are :

• Development tests: Unit Tests and Integration tests. They must be automated to prevent defects in the code, facilitate changes, decrease dependency between classes and improve the software design.
• Acceptance Tests : basic requirements from customer.
• API level: functional tests of the system in the level of API, without GUI dependence. It identifies business rules failures.
• GUI Level : the GUI must be tested when the system interface is stable and will not change.
• Non Functional Tests:  Security Tests, Performance tests, Stress and Volume tests, they must be performed when the system funcionalities are completed and can simulate the real production environment.
• Failures found: The critical or major failures found during exploratory test execution are important candidates to be automated to ensure that the failure will not return to the system after code changes.
• Repetitive tests: The tests that have to be executed many times have to me automated.

When to Automate Testing?

It's better to automate tests when ALL project team are envolved, commited in the automation activities and conscious of the importance and benefits of this activity. Everybody must to understand that it requires time to be spent and all should be willing to help. The automation must be planned and in the project schedule. With this, the automation can be sucessful.

When the automation activity is delegated to the responsibility of one person or one team it gets more difficult communication process and synchronization of tasks.

Common Risks of Test Automation [2]:

• Unrealistics expectations: it is not possible to automate 100% of the tests and manual tests still must be performed.
• Maintenance cost: depending of the system platform, the cost of maintenance of the test automation can be expensive and requires time when the system requirements change.
• Technical difficulties: when the system has a complex platform and lack of testing tools adequated and have no time to build one, it is not worth to automate tests.
• Lack of development process: without a development process organized, customer requirements and architecture documentation, it is not possible to automate the test activities. 
• Lack of management support: the management of the organization must to know the value of test automation activity and the benefits for the project. Many projects has no test automation because the manager think that test automation is expensive and it's not agregate value to build the system. The old idea of the just development code is important to build a system.

References

[1] Dustin, et al , Automated Software Testing, 1999.

[2]  Mark Fewster and Dorothy Graham, Software Test Automation, 2007.

INTRODUCTION

Hello All,

In my first post I will introduce myself and why I created this blog.

My name is Eliane Collins, I am a Computing Engineer and I live in Manaus, Brazil.

Now, many people must to ask: Where is Manaus?

See the red ballloon in the picture below.

Picture from GoogleMaps

Manaus is located in the middle of Amazonia. It is a industrial city, The population in 2012 was 1.85 million people. The Brazilian government offers incentive to companies located here. Local industries include brewing, shipbuilding, soap manufacturing, the production of chemicals, computers, motorcycles and petroleum refining ofoil brought in by barge and tourism. The mobile phone companies Nokia, Siemens, Sumsung, LG, Sagem and Gradiente operate mobile phone manufacturing plants in Manaus. For this reason a lot of software companies are here. (http://en.wikipedia.org/wiki/Manaus)

And finally a little bit about me:

My graduation is in Computing Engineering on University of State of Amazonas, I am actually master degree student in the University Federal of Amazonas.

I work in Software development since 2004 and specifically in Software Testing since 2005. I am ISTQB Certified Tester (CTFL). My experience include Software Testing Process, Software Testing Automation, Software Testing for Mobile Phones, Software Web Development JavaEE, JavaME, Windows Phone and Software Development for Digital TV.

I used to work for Siemens/Benq Mobile as Testing Analyst and nowadays I work as Testing Coordinator at Nokia Technology Institute in Manaus.

 I am also instructor of a course of test automation and I writed some papers since 2010 relating my professional experiences.  My papers in 2012:

• AST 2012 : Software Test Automation Practices in Agile Development Environment: An Industry Experience Report.
• STA 2012 : Strategies for Agile Software Testing Automation: An Industrial Experience.
• ICGSE 2012 :An Industrial Experience on the Application of Distributed Testing in an Agile Software Development Environment. 
• Brazillian Symposium of Software Quality (SBQS 2012): Aplicando Testes Ágeis com Equipes Distribuídas: Um Relato de Experiência

For more professional information my profile is on linkedIn.

And Why to create this blog?

I have 3 key reasons:

First because to automate tests is one of the activities that I most enjoy doing in my job. Second because throughout my career in software testing I went through several experiences automating tests for different kinds of software and thirdly because in presentations and lectures that I've done many professionals describe the difficulties of test teams to accomplish the automation activities during their projects. For these reasons I think my humble knowledge and experience in this subject can help software development teams. :)

I hope you enjoy the next posts! Thank you.