Get Trained and start executing UFT tests from Jenkins. Click Here for more details.

Hello Friends,

In this post, we will discuss integrating Jenkins CI tool with HP UFT.

Introduction

Jenkins is a CI tool which basically monitors external job executions and it’s mainly used by developers for building and deployment of code. It provides several functions like scheduling of tasks called jobs in several remote machines called as slaves in terms of Jenkins vocabulary. In this post, we are going to learn how to setup Jenkins from scratch for executing UFT automated test cases.

First, we need to understand, what is the need for running the automated tests with CI tools? It’s not just applicable for Jenkins, its applicable for any other CI tools or automation tools available in the market. Most of the people are going to tell you about agile and Smoke testing, those are correct and we will go back to this topic in other posts. First, we will look into more common issues which teams will face.

I was very happy with learning new languages and implementing new things for my project, and then one day we did some analysis and it seemed that we were spending most of our time in executing older regression test suites and we have only 20-30 % of our time left for development of new test cases, and it’s the problem which was growing day by day more severe.

So, the only solution for this problem is to assign the execution of the scripts to the actual owners. It seemed ideal, but in the real world that’s not as easy as we think, everybody has their own tasks to be completed, plus they didn’t have the knowledge for using UFT and you will be pushed back to ground zero, from where you started.

On the other side, tests we had were automated test cases, and hence by definition, they are supposed to be run automatically. But due to the manual interventions required due to the nature of scripts or application, it was close to impossible to execute the most of the scripts on a headless remote machine.

To resolve these issues we tried HP ALM’s Lab management tool, but I realized very soon that it can only execute the test cases so if you have to execute a bunch of scripts before or after execution of the scripts, you cannot. Also, there is no way to approve the results and a lot of time issues with the remote machine cause the test to fail.

There is another way of doing it using Automation Object model, but I really don’t want to take the headache of maintaining another piece of code which might change between versions of UFT.

I tried Jenkins and I am still using it, basically, it does everything, let’s see how…

Installation of Jenkins Server

If you don’t have Jenkins server, then follow this guide for the installation of the server. If you have installed Jenkins server using windows setup, you can access Jenkins server using the following URL

http://ServerName:8080

Check your Jenkins workspace path and update if required.

Create slave machine for the execution of the UFT scripts(Machine where UFT is installed), and install the following plug-ins

List of required plugins for the setup

• Copy Artifact Plugin
• Dashboard View Plugin
• Discard Old Build Plugin
• Email Extension Plugin
• Extra Columns Plugin
• Flexible Publish Plugin
• HP Application Automation Tools Plugin
• Test Result Analyzer Plugin
• Time Stamper Plugin
• Workspace Cleanup Plugin

Most of them are self-explanatory, but if you need any other details then, Google with the exact name of the plugin.

Creating Jenkins job

Once, you are able to open the Jenkins server, Click on “New Item”

Provide any Item name, select “freestyle project” and click on “OK” button

Once, you click on OK button, New Job configuration page will open. This is the main configuration page, where we will provide all of the configuration details about how the Job should execute.

Fill out the following information within the job configuration page

1) Description: Description about the job and the UFT test which is tied to this job.

2) Check on “Restrict where this project can be run” and provide the slave machine name with UFT installed. This is a very important step, because otherwise Jenkins will choose a slave machine randomly, and might choose Jenkins master machine (where Jenkins is installed) for execution. This will cause the job to fail.

3) Select appropriate build triggers, if you are using any source code control for storing UFT tests (Except ALM) e.g. GIT, SVN, TFS.

4) Under Build environment select “delete workspace before build starts”. This will help us in cleaning up the workspace and have a fresh start for each execution.

5) Click on Add build step button under build and select “Execute HP Tests from file system”. This will allow you to execute the tests from file system e.g. “C:\” or network drives.

6) Provide UFT tests or Solution path in the “Tests” text box. If you provide UFT’s solution file path then all tests under solution will be executed one after another. Leave Time out seconds blank.

7) Click on “Add post build action” under post build actions, and select “Publish HP test results”. Select “Always Archive HP test Reports”. This step will generate and archive Junit results for the execution on the Jenkins server, which can be used for analysis. we will discuss this in detail later.

8) If you have configured the email server then Add another post build step and “Editable email notification”. This option will send automated emails after the execution is over. We can attach any files in the email or we can send any files as the part of the email body.

After filling out the above information, Click on Save button and start the job by clicking on “Build now” button within job details page.

Once the execution is started, click on Build number, under build history, to check the current execution details

While the test is executing, you can click on Console output, to check the current execution status.

Once the execution is completed, you can see the test results on the Build page, under test results

This completes the basic configuration of the Jenkins Job.

How it works?

When you execute a UFT test from Jenkins, HP automation tools plugin will copy the following files to the Jenkins Job workspace directory

1) HpToolsLauncher.exe – This is the file used for actual execution of the UFT tests .when you take a closer look at the console output of the Jenkins build, you will see a line

"C:\Jenkins\workspace\JENKINS UFT TEST\HpToolsLauncher.exe" -paramfile props23032015132505096.txt

If you execute this line on the slave machine UFT test will execute.

2) LRAnalysisLauncher.exe – For load runner.

3) propsXXXXX.txt – This is the properties file for the HP Automation tools plugin. This will store different parameters required for the execution of UFT/Load tests.

4) ResultsXXXXX.xml – This is the JUnit results generated by the HpToolsLauncher.exe after the execution is completed. When we choose to create a post build step and choose to “Publish HP run results”. This Junit file will be uploaded back to the Jenkins and produce the test results shown in step 12 above.

General Issues faced during test execution

There are several issues you may encounter during the execution of UFT tests from Jenkins

1) If scripts have any “MsgBox” statement or any “InputBox” statement, scripts will directly fail.

2) If the machine is unlocked, then there is a chance for complex applications to fail.

3) We have to make sure that machine is cleaned up before and after the execution of tests(Closing all browsers and any other types of files like excel sheets or word documents).

4) We have to make sure that UFT scripts have a proper error handling mechanism or UFT will throw an error message and Jenkins Job will fail as soon as Popup appears on the page.

5) If we have a proper framework for our scripts, then most likely we will be executing multiple tests within one UFT test. Because Jenkins HP Automation tool can only count UFT tests and there is no way to indicate no. of tests, if you execute 2 UFT tests having 20 tests each, Jenkins dashboard will only display 2 tests executed, not 40 tests executed.

6) If UFT test report’s a failure in any of the tests, you will see the failure of Jenkins complete Job instead of a particular test case.

let’s see how to resolve each of these issues to create a perfect Jenkins job for executing automated UFT tests.

1) Replace any statement which requires human intervention

We must scan our scripts and replace any statements which require human intervention like MsgBox statements must be commented and should only be used during debugging. For any steps which use Input boxes for accepting inputs, use test parameters and action parameters. For receiving information back, use reporter.reportevent instead of msgbox statements.

2) Use Physical desktops, instead of virtual machines

If we use virtual machines as Jenkins slave machines, then we will have to make sure that the machines are up and running all the time, because virtual machine can go blank without any prior warning, if the user disconnects them.

3) Unlocking remote slave machine before executions

To overcome this issues, if possible change the machine setting, so that machine never locks automatically. But this option is not available in most of the companies because the machines running UFT will most probably have admin rights and it’s not advisable to leave the machine with admin rights unlocked.

First test your scripts, if they are able to run in the locked machine. If not then, you can use the following commands to unlock the Physical desktop machine from another computer (In case you are at home or other location).

For Unlocking the Remote Desktop(Physical machine)

Run on the command prompt

Query User

This command will provide the Session ID for your Login Session

Once you have the session id of your session execute the following command

Tscon <Session id from previous commapn> /DEST:Console

This command will unlock the desktop machine and you can execute your test normally.

4) Include Machine Cleanup Framework in UFT

Machine cleanup framework is explained here. This framework will use the VBScript classes for executing predefined cleanup code. This is required because, if the previous execution of code leaves some things open then, your subsequent executions might fail.

5) Include proper error handling mechanism in your scripts

Recovery scenarios cannot handle errors properly, for running tests from Jenkins you will need a way to handle all kinds of error, including UFT’s general run error’s. Because, if UFT throws a general run error, then your Jenkins execution will fail, while UFT will be waiting for someone to click on the error and close the test manually and this will create issues with the subsequent test executions. A detailed description of how to implement this framework is explained here.

Executing tests in Parallel

For execution multiple tests in parallel, you can choose to do one of these two methods

1) Open the tests and update the Label expression under “Restrict where this project can be run”, with the slave machine name. Once you update the two different tests with different machine names, you can execute the tests in parallel on those two machines (provided you have the proper licenses for executions).

2) Now, the previous method works, but that’s not the best way to execute parallel tests within Jenkins. To implement the second method you will have to figure out the machines with similar properties like “UFT MACHINE WITH IE 8”, so this is a specific machine having UFT installed and can be used for the apps which are only compatible with IE 8. Once you have all the machine characteristics identified you can then open the slave configurations for the machine and update these names in the Slave labels. Once this Logical grouping of the Machines is done then you can provide this logical name with “Restrict where this project can be run”, instead of machine names. Using this method, you wont have to actually remember about the machine names and go through the whole hard process to identifying which machine you should execute the test.

One more advantage of this method is if you have two machine with Label “UFT MACHINE WITH IE 8”, and when you execute the test, if the one of the machine is busy, then Jenkins will automatically Pickup the second machine and use it for execution.

Building test queue(sequential test execution)

Jenkins will automatically build the execution queue, once you select more number of jobs then the machines you have. These tests will be executed automatically when the machine will become free. To check this, Update the “Restrict where this project can be run” of three Jobs with the same slave name. Now, click on home page and Click on “Schedule a build” button for all three jobs, one after another and you will see the build queue, on the left side of the home page under “Build queue”.

Executing tests from ALM

For running tests from ALM, you have to make three changes to the current configuration explained above.

1) Add ALM Server

open Jenkins -> Manage Jenkins -> Configure System and find the sub section “Application Lifecycle Management”.

Click on “Add ALM Server” button. Provide the server name as Test or Prod or Dev and enter the server URL till qcbin. Click on save and apply.

You can add multiple ALM Servers.

2) Use “Execute HP Functional test from HP ALM” instead of “Execute HP tests from file system”.

Add the Build Step “Execute HP Functional test from file system”. Select ALM Server name from the “ALM Server” combobox. Provide Username, password, Domain, project and Test Set Path for the execution of the tests. Test set path is the actual physical path which you see in the ALM test lab tree structure starting with Root\

3) Check the advanced setting,

if you want to execute the tests on specific host defined in ALM.

You will see the following changes in the execution process

Modified Console Output

When you execute your tests from ALM Server, you will see one change in the Console output, you will start seeing all of the Reporter.reportevent statements on the Jenkins console log.

This feature can be used to check the current progress of the execution.

Automatic Execution update in ALM

Test execution status will be uploaded automatically to ALM Server.

Additional helpful setting\upgrades

1) Instead of using “Publish HP Run results option”, Create and upload your our own Junit results

When, one UFT test executes 100 test cases, HP Automation tools plugin will only return either one pass or one fail. To over come this create your own Junit results xml file at runtime after the UFT execution is completed. This will also serve as a checkpoint, if the UFT execution is completed successfully then only we should have a JUnit file generated.

Once you have this functionality implemented, you will start seeing awsome results on Jenkins and you will unlock the real power of Jenkins. JUnit results XML will look like this

<testsuites>
<testsuite Name=“FirstAutomation” test=“2” failures=“0” time=““>
<testcase name=“test1” classname=“FirstAutomation.TestCases” time=“27.68359” status=“pass”>
<testcase name=“test2” classname=“FirstAutomation.TestCases” time=“13.33203” status=“pass”>
</testsuite>
</testsuites>

2) Upload test Artifacts to Jenkins

Jenkins will allow to upload the artifacts back to the Jenkins server. We have to use the copy artifacts plugin to do this. Within UFT you can create a function and call it within the Machine cleanup Framework’s class initialization(to make sure it executes) to setup the export of UFT datatable and UFT run results to the Jenkins workspace like this

Public Function SetQTPParameters()
Dim App 'As Application
Set App = CreateObject(“QuickTest.Application”)
App.Options.Run.AutoExportReportConfig.AutoExportResults = True
App.Options.Run.AutoExportReportConfig.StepDetailsReport = True
App.Options.Run.AutoExportReportConfig.DataTableReport = True
App.Options.Run.AutoExportReportConfig.LogTrackingReport = False
App.Options.Run.AutoExportReportConfig.ScreenRecorderReport = False
App.Options.Run.AutoExportReportConfig.SystemMonitorReport = False
App.Options.Run.AutoExportReportConfig.ExportLocation = JenkinsWorkSpacePath
App.Options.Run.AutoExportReportConfig.UserDefinedXSL = ““
App.Options.Run.AutoExportReportConfig.StepDetailsReportType = “HTML”
App.Options.Run.AutoExportReportConfig.StepDetailsReportFormat = “Detailed”
App.Options.Run.AutoExportReportConfig.ExportForFailedRunsOnly = False
End Function