Guide Automation & DeviceFarm HU Part 2

Part 2: Execute Script with Tester 4.0

Tester 4.0: Acts as a Runner or Scheduler to execute the test scripts on many devices.

I. Execute a Script and check the test result

1. Run Now

Step 1: Click on the button [Run Test Script Now] -> The popup Run Test Script will be displayed

Step 2: Select Device (In the Run Test Script popup, select one or more Android devices)-> Click on the [Next] button

Step 3: Select Test Script (After selecting Android devices, the popup list of Android test scripts will be displayed –> select one, or more scripts that the users want to run –> Then click on the [Next] button)

Step 4: Select test language(s)

Input a Job name and select test language(s)
Click on the button [Run Test Now]

Step 5: View running test script(s) (Show the correct number of devices and the name of each device selected for running from step 2 -> Click on the [View] button next to each device name to show the streaming screen of that device)

Step 6: After the test script execution is finished: The Test Report will be shown in the Project Detail screen. To view the detail of a Test report: click on each test report item in the list

2. Run a script On Schedule

Step 1: Click on the [Add Schedule Job] button

➩ The popup select devices & test scripts (similar to the “Run Now” feature) will be displayed

Step 2: Setting the time and date to run test scripts automatically:

Input the Job name
Select test language(s)
Select a time to run test scripts: including the time and the day of the week
Click on the [Add Schedule Test] button

➩ After finish adding a Schedule, at the scheduled time, the selected test scripts will be executed and the test result will be saved.

3. Check the test result

After the test script execution is finished, the Test Report will be shown in the Project Detail screen.

To view the detail of a Test report, please click on each test report item in the list.

User wants to review detail report of test script, user clicks on that test script. Detail report will be displayed.

There are 3 types of results: Pass, Fail and Warning.

Pass: Script works, no script errors, test script return result exactly as expected.

Fail: There was a script error or an input error, not getting results fromTester 4.0. Or there is no script error but Tester 4.0 returns resultdifferent from expectation. If the result is fail, the system will displaydetail of error and error step.

If user use AI to check the expectation in test script, it will have more result: Warning. The script works, there are no script errors, the AI returns the results, but the AI’s result is that something is wrong. Specifically: An element’s image comparison function returns FALSE. When the result is warning, the user needs to confirm the result.

AI when comparing 2 screen shots will give the same ratio of 2 images, the value runs from 0 to 100%. Tester 4.0 defines a threshold level, if it is below this level, it is considered a problem. For example, if threshold setting is 95%:

– A.I returns 2 images that have similarity greater than or equal to 95% -> no problem. Tester 4.0 will return the result – Pass.

– A.I returns 2 images with the same 95% or less -> considered “problematic”. Tester 4.0 will return the result – Warning.

At the moment, the threshold setting is 95%.

When the result is Warning, user needs confirmation by follow steps:

Step1: Clicking on icon then dropdown list will be shown.

Step 2: If user wants to confirm OK, please click on [Confirm OK] button. Popup Note for this confirmation will be displayed. User inputs the reason then clicks on [Confirm] button. The result of test script will change from warning to pass.

Step 3: If user wants to confirm NG, please click on [Confirm NG] button. Popup Note for this confirmation will be displayed. User inputs the reason then clicks on [Confirm] button. The result of test script will change from Warning to Failed.