Executing automated UFT\ALM tests from Jenkins

Get Trained and start executing UFT tests from Jenkins. Click Here for more details.

Hello Friends,

In this post, we will discuss integrating Jenkins CI tool with HP UFT.

Introduction

Jenkins is a CI tool which basically monitors external job executions and it’s mainly used by developers for building and deployment of code. It provides several functions like scheduling of tasks called jobs in several remote machines called as slaves in terms of Jenkins vocabulary. In this post, we are going to learn how to setup Jenkins from scratch for executing UFT automated test cases.

First, we need to understand, what is the need for running the automated tests with CI tools? It’s not just applicable for Jenkins, its applicable for any other CI tools or automation tools available in the market. Most of the people are going to tell you about agile and Smoke testing, those are correct and we will go back to this topic in other posts. First, we will look into more common issues which teams will face.

I was very happy with learning new languages and implementing new things for my project, and then one day we did some analysis and it seemed that we were spending most of our time in executing older regression test suites and we have only 20-30 % of our time left for development of new test cases, and it’s the problem which was growing day by day more severe.

So, the only solution for this problem is to assign the execution of the scripts to the actual owners. It seemed ideal, but in the real world that’s not as easy as we think, everybody has their own tasks to be completed, plus they didn’t have the knowledge for using UFT and you will be pushed back to ground zero, from where you started.

On the other side, tests we had were automated test cases, and hence by definition, they are supposed to be run automatically. But due to the manual interventions required due to the nature of scripts or application, it was close to impossible to execute the most of the scripts on a headless remote machine.

To resolve these issues we tried HP ALM’s Lab management tool, but I realized very soon that it can only execute the test cases so if you have to execute a bunch of scripts before or after execution of the scripts, you cannot. Also, there is no way to approve the results and a lot of time issues with the remote machine cause the test to fail.

There is another way of doing it using Automation Object model, but I really don’t want to take the headache of maintaining another piece of code which might change between versions of UFT.

I tried Jenkins and I am still using it, basically, it does everything, let’s see how…

Installation of Jenkins Server

If you don’t have Jenkins server, then follow this guide for the installation of the server. If you have installed Jenkins server using windows setup, you can access Jenkins server using the following URL

http://ServerName:8080

Check your Jenkins workspace path and update if required.

Create slave machine for the execution of the UFT scripts(Machine where UFT is installed), and install the following plug-ins

List of required plugins for the setup

  • Copy Artifact Plugin
  • Dashboard View Plugin
  • Discard Old Build Plugin
  • Email Extension Plugin
  • Extra Columns Plugin
  • Flexible Publish Plugin
  • HP Application Automation Tools Plugin
  • Test Result Analyzer Plugin
  • Time Stamper Plugin
  • Workspace Cleanup Plugin

Most of them are self-explanatory, but if you need any other details then, Google with the exact name of the plugin.

Creating Jenkins job

Once, you are able to open the Jenkins server, Click on “New Item”

Jenkins New Job

Provide any Item name, select “freestyle project” and click on “OK” button

Jenkins New Job Name

Once, you click on OK button, New Job configuration page will open. This is the main configuration page, where we will provide all of the configuration details about how the Job should execute.

Fill out the following information within the job configuration page

1) Description: Description about the job and the UFT test which is tied to this job.

2) Check on “Restrict where this project can be run” and provide the slave machine name with UFT installed. This is a very important step, because otherwise Jenkins will choose a slave machine randomly, and might choose Jenkins master machine (where Jenkins is installed) for execution. This will cause the job to fail.

3) Select appropriate build triggers, if you are using any source code control for storing UFT tests (Except ALM) e.g. GIT, SVN, TFS.

4) Under Build environment select “delete workspace before build starts”. This will help us in cleaning up the workspace and have a fresh start for each execution.

5) Click on Add build step button under build and select “Execute HP Tests from file system”. This will allow you to execute the tests from file system e.g. “C:\” or network drives.

Jenkins Execute HP tests From File System

6) Provide UFT tests or Solution path in the “Tests” text box. If you provide UFT’s solution file path then all tests under solution will be executed one after another. Leave Time out seconds blank.

7) Click on “Add post build action” under post build actions, and select “Publish HP test results”. Select “Always Archive HP test Reports”. This step will generate and archive Junit results for the execution on the Jenkins server, which can be used for analysis. we will discuss this in detail later.

8) If you have configured the email server then Add another post build step and “Editable email notification”. This option will send automated emails after the execution is over. We can attach any files in the email or we can send any files as the part of the email body.

After filling out the above information, Click on Save button and start the job by clicking on “Build now” button within job details page.

Jenkins Build Now

Once the execution is started, click on Build number, under build history, to check the current execution details

Jenkins Build Number

While the test is executing, you can click on Console output, to check the current execution status.

Jenkins Console Output

Once the execution is completed, you can see the test results on the Build page, under test results

Jenkins test output

This completes the basic configuration of the Jenkins Job.

How it works?

When you execute a UFT test from Jenkins, HP automation tools plugin will copy the following files to the Jenkins Job workspace directory

1) HpToolsLauncher.exe – This is the file used for actual execution of the UFT tests .when you take a closer look at the console output of the Jenkins build, you will see a line

"C:\Jenkins\workspace\JENKINS UFT TEST\HpToolsLauncher.exe" -paramfile props23032015132505096.txt

If you execute this line on the slave machine UFT test will execute.

2) LRAnalysisLauncher.exe – For load runner.

3) propsXXXXX.txt – This is the properties file for the HP Automation tools plugin. This will store different parameters required for the execution of UFT/Load tests.

4) ResultsXXXXX.xml – This is the JUnit results generated by the HpToolsLauncher.exe after the execution is completed. When we choose to create a post build step and choose to “Publish HP run results”. This Junit file will be uploaded back to the Jenkins and produce the test results shown in step 12 above.

General Issues faced during test execution

There are several issues you may encounter during the execution of UFT tests from Jenkins

1) If scripts have any “MsgBox” statement or any “InputBox” statement, scripts will directly fail.

2) If the machine is unlocked, then there is a chance for complex applications to fail.

3) We have to make sure that machine is cleaned up before and after the execution of tests(Closing all browsers and any other types of files like excel sheets or word documents).

4) We have to make sure that UFT scripts have a proper error handling mechanism or UFT will throw an error message and Jenkins Job will fail as soon as Popup appears on the page.

5) If we have a proper framework for our scripts, then most likely we will be executing multiple tests within one UFT test. Because Jenkins HP Automation tool can only count UFT tests and there is no way to indicate no. of tests, if you execute 2 UFT tests having 20 tests each, Jenkins dashboard will only display 2 tests executed, not 40 tests executed.

6) If UFT test report’s a failure in any of the tests, you will see the failure of Jenkins complete Job instead of a particular test case.

let’s see how to resolve each of these issues to create a perfect Jenkins job for executing automated UFT tests.

1) Replace any statement which requires human intervention

We must scan our scripts and replace any statements which require human intervention like MsgBox statements must be commented and should only be used during debugging. For any steps which use Input boxes for accepting inputs, use test parameters and action parameters. For receiving information back, use reporter.reportevent instead of msgbox statements.

2) Use Physical desktops, instead of virtual machines

If we use virtual machines as Jenkins slave machines, then we will have to make sure that the machines are up and running all the time, because virtual machine can go blank without any prior warning, if the user disconnects them.

3) Unlocking remote slave machine before executions

To overcome this issues, if possible change the machine setting, so that machine never locks automatically. But this option is not available in most of the companies because the machines running UFT will most probably have admin rights and it’s not advisable to leave the machine with admin rights unlocked.

First test your scripts, if they are able to run in the locked machine. If not then, you can use the following commands to unlock the Physical desktop machine from another computer (In case you are at home or other location).

For Unlocking the Remote Desktop(Physical machine)

Run on the command prompt

Query User

This command will provide the Session ID for your Login Session

JenkinsQueryUser

Once you have the session id of your session execute the following command

Tscon <Session id from previous commapn> /DEST:Console

This command will unlock the desktop machine and you can execute your test normally.

4) Include Machine Cleanup Framework in UFT

Machine cleanup framework is explained here. This framework will use the VBScript classes for executing predefined cleanup code. This is required because, if the previous execution of code leaves some things open then, your subsequent executions might fail.

5) Include proper error handling mechanism in your scripts

Recovery scenarios cannot handle errors properly, for running tests from Jenkins you will need a way to handle all kinds of error, including UFT’s general run error’s. Because, if UFT throws a general run error, then your Jenkins execution will fail, while UFT will be waiting for someone to click on the error and close the test manually and this will create issues with the subsequent test executions. A detailed description of how to implement this framework is explained here.

Executing tests in Parallel

For execution multiple tests in parallel, you can choose to do one of these two methods

1) Open the tests and update the Label expression under “Restrict where this project can be run”, with the slave machine name. Once you update the two different tests with different machine names, you can execute the tests in parallel on those two machines (provided you have the proper licenses for executions).

2) Now, the previous method works, but that’s not the best way to execute parallel tests within Jenkins. To implement the second method you will have to figure out the machines with similar properties like “UFT MACHINE WITH IE 8”, so this is a specific machine having UFT installed and can be used for the apps which are only compatible with IE 8. Once you have all the machine characteristics identified you can then open the slave configurations for the machine and update these names in the Slave labels. Once this Logical grouping of the Machines is done then you can provide this logical name with “Restrict where this project can be run”, instead of machine names. Using this method, you wont have to actually remember about the machine names and go through the whole hard process to identifying which machine you should execute the test.

JenkinsSalveConfig

JenkinsRestrictWhereProjectsCanRun

One more advantage of this method is if you have two machine with Label “UFT MACHINE WITH IE 8”, and when you execute the test, if the one of the machine is busy, then Jenkins will automatically Pickup the second machine and use it for execution.

Building test queue(sequential test execution)

Jenkins will automatically build the execution queue, once you select more number of jobs then the machines you have. These tests will be executed automatically when the machine will become free. To check this, Update the “Restrict where this project can be run” of three Jobs with the same slave name. Now, click on home page and Click on “Schedule a build” button for all three jobs, one after another and you will see the build queue, on the left side of the home page under “Build queue”.

Executing tests from ALM

For running tests from ALM, you have to make three changes to the current configuration explained above.

1) Add ALM Server

open Jenkins -> Manage Jenkins -> Configure System and find the sub section “Application Lifecycle Management”.

Click on “Add ALM Server” button. Provide the server name as Test or Prod or Dev and enter the server URL till qcbin. Click on save and apply.

You can add multiple ALM Servers.

2) Use “Execute HP Functional test from HP ALM” instead of “Execute HP tests from file system”.

Add the Build Step “Execute HP Functional test from file system”. Select ALM Server name from the “ALM Server” combobox. Provide Username, password, Domain, project and Test Set Path for the execution of the tests. Test set path is the actual physical path which you see in the ALM test lab tree structure starting with Root\

JenkinsExecuteTestFromALMServer

3) Check the advanced setting,

if you want to execute the tests on specific host defined in ALM.

You will see the following changes in the execution process

Modified Console Output

When you execute your tests from ALM Server, you will see one change in the Console output, you will start seeing all of the Reporter.reportevent statements on the Jenkins console log.

This feature can be used to check the current progress of the execution.

Automatic Execution update in ALM

Test execution status will be uploaded automatically to ALM Server.

Additional helpful setting\upgrades

1) Instead of using “Publish HP Run results option”, Create and upload your our own Junit results

When, one UFT test executes 100 test cases, HP Automation tools plugin will only return either one pass or one fail. To over come this create your own Junit results xml file at runtime after the UFT execution is completed. This will also serve as a checkpoint, if the UFT execution is completed successfully then only we should have a JUnit file generated.

Once you have this functionality implemented, you will start seeing awsome results on Jenkins and you will unlock the real power of Jenkins. JUnit results XML will look like this

<testsuites>
<testsuite Name=“FirstAutomation” test=“2” failures=“0” time=““>
<testcase name=“test1” classname=“FirstAutomation.TestCases” time=“27.68359” status=“pass”>
<testcase name=“test2” classname=“FirstAutomation.TestCases” time=“13.33203” status=“pass”>
</testsuite>
</testsuites>
When you publish this file after one UFT test execution you will see two passed test instead of one passed tests.
JenkinsJunitUploadedResults
In this above example, i have executed several tests and 17 of them failed. All of the tests were under one UFT test
To publish the JUnit results, add the post build step “Publish Junit test result report”. Provide the JUnit file path and enter 1.0 in the Health report amplification factor.

2) Upload test Artifacts to Jenkins

Jenkins will allow to upload the artifacts back to the Jenkins server. We have to use the copy artifacts plugin to do this. Within UFT you can create a function and call it within the Machine cleanup Framework’s class initialization(to make sure it executes) to setup the export of UFT datatable and UFT run results to the Jenkins workspace like this

Public Function SetQTPParameters()
Dim App 'As Application
Set App = CreateObject(“QuickTest.Application”)
App.Options.Run.AutoExportReportConfig.AutoExportResults = True
App.Options.Run.AutoExportReportConfig.StepDetailsReport = True
App.Options.Run.AutoExportReportConfig.DataTableReport = True
App.Options.Run.AutoExportReportConfig.LogTrackingReport = False
App.Options.Run.AutoExportReportConfig.ScreenRecorderReport = False
App.Options.Run.AutoExportReportConfig.SystemMonitorReport = False
App.Options.Run.AutoExportReportConfig.ExportLocation = JenkinsWorkSpacePath
App.Options.Run.AutoExportReportConfig.UserDefinedXSL = ““
App.Options.Run.AutoExportReportConfig.StepDetailsReportType = “HTML”
App.Options.Run.AutoExportReportConfig.StepDetailsReportFormat = “Detailed”
App.Options.Run.AutoExportReportConfig.ExportForFailedRunsOnly = False
End Function

After creating this function, on the Jenkins configuration, add a build step “Archive the artifacts”.  This is Upload the results back to the Jenkins server and will be visible for each execution. The default Archival path is Jenkins workspace, but we can also point it to the child folder inside jenkins workspace.

e.g. If Jenkins workspace path is “C:\Jenkins”(Create this in Slave Config -> Remote Root Directory) then the Job execution path will be “C:\Jenkins\JENKINS UFT TEST\”

to archive the artifacts from the child folder within workspace e.g. “C:\Jenkins\JENKINS UFT TEST\ExecutionResults”

use “ExecutionResults/*” in the files to archive textbox

In addition Upload any artifacts related to the test execution to Jenkins, if required.

3) Use Dashboard Plugin to create execution dashboards

Once, you have dashboard plugin installed in the Jenkins, create dashboard views and add test execution statistics Grid to it. This will pull the data from the Jobs latest JUnit files and publish the data on the Dashboard page Providing summary on the Dashboard home page.

JenkinsResultsDashboard

Above is the snippet of the Test execution statistics Grid.

4) Use Flexible publish to produce the results

Use the flexible publish plugin to figure out things like, if the files are generated before actually passing the Jenkins job or failing the Jenkins Job. One example is check if the Junit results are generate before archiving them to Jenkins server. If the file is absent then Jenkins will throws an error and fail the job.

5) Close and open HP Remote agent during each run session

I have included a separate batch file for running the remote agent launcher during each run session and close it after the execution is over. This will make sure that all of the UFT related processes and connection are re-established during each run session.

6) Use “test result analyzer” for result analysis

You can use the test result analyzer for analyzing the results of two separate executions. To Open, Click on “Test Result Analyzer” on the job description page.

Test Results Analyzer

If, we have 100’s of tests, it might not be very easy to analyze the results between two executions, but if you have Custom JUnit reports generated, then you can use this feature for analyzing the results between any two executions

7) Use Batch File to Kill UFT

Create a batch file to kill UFT and add this to build step after the step for the execution of the UFT Test. This will make sure that UFT is not hanged in between and also if the Jenkins Job is stopped due to some reason you will have the machine ready for the next one.

Let me know, if you have any questions and i will try to resolve them.

Thanks

Advertisements

257 responses to “Executing automated UFT\ALM tests from Jenkins

  1. Hi,
    When i run in UFT 12.02 , i am unable to see the results. Will it support UFT 12.02 with the above steps.

    Like

      • Thanks Summet, for the information , can i get some help on junit result generation. As with 12.02 i am not getting uft results/reports. its saying to install later version of uft or run results viewer later version.

        Like

  2. Hi,

    I’m unable to run it successfully. Can you please help? Below is the error.

    “Started…”
    Launcher timeout is 10675199::02:48:05
    Controller Polling Interval: 30 seconds
    PerScenarioTimeout: 00::00:10:00 minutes
    FileSystemTestRunner timeout is 10675199.02:48:05.4775807
    Mc connection info is – Mc HostAddress: , McPort: 8080, Username: , UseSSL: 0, UseProxy: 0, ProxyType: 0, ProxyAddress: , ProxyPort: 0, ProxyAuth: 0, ProxyUser:
    1 tests found:
    C:\Users\****\UFT\DPAS_Letter_CS05
    ============================================================================
    26/07/2017 14:53:24 Running: C:\Users\****\UFT\DPAS_Letter_CS05
    Test result: Error
    26/07/2017 14:53:26 Test complete: C:\Users\****\UFT\DPAS_Letter_CS05
    ——————————————————————————————————-
    ================================================
    Run status: Job failed, total tests: 1, succeeded: 0, failures: 0, errors: 1
    Error : C:\Users\****\UFT\DPAS_Letter_CS05
    ================================================
    Build step ‘Execute HP tests from file system’ changed build result to FAILURE
    Recording test results
    Report archiving mode is set to: ONLY_ARCHIVE_FAILED_TESTS_REPORT
    Finished: FAILURE

    Like

    • Do you see anything executing on screen ? Have u used Inputbox statement in your script ? When u execute the script from UFT what’s the result ?

      Can u execute a sample UFT script from Jenkins successfully?

      Please let me know.

      Thanks,
      Sumeet

      Like

      • Hi,

        The same script successfully runs from UFT where as it is unsuccessful in Jenkins. It is not executing anything. When the build is triggered, it fails in couple of seconds.
        I have tried a new script as well. Still no luck. Same behaviour- From UFT it is successfully running whereas it fails in Jenkins with above error.

        Also I tried to configure Jenkins to run it from ALM. Still no luck. I’m facing the following error.

        “Started…”
        Timeout is set to: -1
        Run mode is set to: RUN_LOCAL
        ============================================================================
        Starting test set execution
        Test set name: Shakeout, Test set id: 2816
        Could not create scheduler, please verify ALM client installation on run machine by downloading and in installing the add-in form: http://alm.aus.thenational.com/qcbin//TDConnectivity_index.html
        Build step ‘Execute HP functional tests from HP ALM’ changed build result to FAILURE
        Recording test results
        RunResultRecorder: didn’t find any test results to record
        Finished: FAILURE

        Thanks,
        Viswa

        Like

  3. Hi

    Thanks for your valuable article.

    Can you please advise how we can I use parameters from propsXXXXX.txt file within UFT code?

    Regards,
    Alankar

    Like

  4. Hi
    I’m new to Jenkins and now im doing basic level of execution.
    And I need your help resolve my issue.
    Issue Description:
    I’m trying to run the .VBS file by using jenkins and getting below error”

    C:\Users\Admin\Desktop\Test_Sample.vbs(3, 2) Microsoft Excel: Microsoft Excel cannot access the file ‘C:\Users\Admin\Desktop\New.xlsx’. There are several possible reasons:

    The file name or path does not exist.
    The file is being used by another program.
    The workbook you are trying to save has the same name as a currently open workbook.

    I have not opened any excel file and path also correct.

    please find below batch command :
    CScript “C:\Users\Amareswara Rao\Desktop\Test_Sample.vbs”

    And in Test_Sample.vbs file having:
    Dim ExcelObj, WbookObj, mysheet
    Set ExcelObj= CreateObject(“excel.application”)
    Set WbookObj= ExcelObj.workbooks.open(“C:\Users\Admin\Desktop\New.xlsx”)
    Set mysheet= ExcelObj.ActiveWorkbook.Worksheets(“Sheet1″)
    ExcelObj.Visible = False
    ExcelRowCount = mysheet.UsedRange.Rows.Count
    For Iterator = 2 To ExcelRowCount
    ModuleName = Ucase(Trim(mysheet.Cells(Iterator,1)))
    ‘MsgBox ModuleName
    ExecutionFlag = Ucase(Trim(mysheet.Cells(Iterator,2)))
    ‘MsgBox ExecutionFlag

    if (moduleName=”Raj”) Then
    Set qtApp = CreateObject(“QuickTest.Application”)
    If qtApp.launched True then
    qtApp.Launch
    End If
    qtApp.Visible = True ‘MarkON/OFF UFT
    qtApp.Open “F:\Jenkins_Jobs\OnCall_Sanity_Automation\DriverScript\Sample”, False
    Set QTPTest=qtApp.Test
    QTPTest.Run ‘Run the Test
    QTPTest.Close
    ‘qtApp.Close
    qtApp.quit
    Set qtApp = Nothing
    else
    MsgBox “Name not exists”
    End If
    Next
    Please help me to resolve the issue.

    Thanks,
    Amar

    Like

  5. Hi Summet,
    Is creating a jenkins slave compulsory for executing UFT scripts?

    I am accessing the UFT scripts from ALM. I added the test set names in HPE plugin.
    But when i run job i am getting below error in console output.

    Started by user Alluru, Narasimha Raju
    Building in workspace /opt/devops/deployment_services/jenkins/aac-pipeline/workspace/AutomationDemo
    [AutomationDemo] $ /opt/devops/deployment_services/jenkins/aac-pipeline/workspace/AutomationDemo/HpToolsLauncher.exe -paramfile props30112017155050593.txt
    Build step ‘Execute HPE functional tests from HPE ALM’ changed build result to FAILURE
    Recording test results
    RunResultRecorder: didn’t find any test results to record
    Finished: FAILURE

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s