Product Guide Automation & DeviceFarm HU

Contents

Studio: Helps to generate test scripts automatically based on users’ interactions with the devices.

I. Add a new project (Tester 4.0)

Login to: https://tester40.atomp.io
When the Dashboard is shown, click on “Add Project”
Pick a name and set a type for the new project
Click on the [Create New Project] button to finish creating a new project

II. Create a Test Precondition

Precondition can be created and used for multiple scripts with the same precondition. Please follow these steps

1. Click on the [Manage Test Precondition] button.

2. Click on the [Add Precondition] button.

3. Select a device to start creating a precondition

4. Select a precondition parent (if applicable).

5. Click on the [Record script] button to switch to the recording mode to create steps for the precondition
After selecting the precondition -> The [Preview script] button will be enabled. You can select the [Preview script] button to run the pre-condition (Follow step 10,11)

6. Select an element from the streaming screen of the device to peform actions.

7. Choose the apppopriate actions to take -> The steps taken are recorded below.

8. Click on the [Stop] button => the [Preview script] button will be enabled.

9. Click on the [Preview script] button to run and view all created steps.

10. Click on the [Run all] button to run all preconditions.

11. Click on the [Run script] button to run the recorded steps one at a time => After running the script, the device’s streaming screen will show the live execution of whatever steps that were recorded. The step status will be displayed at the end of each step.

12. Click on the [Back] button => the “Create script” screen will be displayed => Click on the [Save] button.

13. Popup “Save as Pre-condition” will be displayed => Enter the information of the precondition that you want to save => Click on the [Save] button => This Precondition will be saved in the “Manage Precondition” screen.

III. Create a new test script

The script creation screen is displayed by step: Click on the [Manage Test Script] item => Click [Create with ATOMP Studio] => Select any device

Upload a CSV file (if necessary)

User can upload a CSV file containing two rows of data (key/value pairs) and use those pairs as variables inside ATOMP Studio.

From the “Predefined Actions” tab, click on “Expand” at “CSV External Input”, then click “Upload” to upload a CSV file.

A CSV file has the following format (Column A: Variable Name, Column B: Value)

After uploading the CSV file, users can use those variables in any textbox while creating test script by typing “$+variable name”

2. [Recording] mode

To start creating a test script, you need to switch to the [Recording] mode by clicking on the button [Record script] or the Eye icon => below is a sample of the Recording mode.

Users can select a screen element (icon, button, text, etc…) to execute some actions.

3. Select Precondition(s)

Step 1: On the script creation screen, please click on the [Add new] button. -> Popup Selected Pre-conditions will be displayed.

Step 2: User can click icon to view detail precondition. Select one or more preconditions from the precondition list of the project.

Step 3: Click [OK]

When the script is running, these selected preconditions will run from top to bottom. List of selected preconditions will be displayed as below:

4. Create Test Steps

Step 1: Switch to the “Select Element” mode (by clicking on the icon )

Step 2: Select an element on the streaming screen

Step 3: Select an action you want to take with the element selected in step 2

If the selected element is an icon or a button, the available actions are (Tap, Double Tap, Long Press)
If the selected element is a Text Input box, click on the [Send Keys] button to input text to the element.

Step 4: If the action is [Send Keys] -> a popup will be shown for users to input characters.

=> After executing 1 action, the tool automatically generates a test script for that action and displays as a step

5. Create an Expectation

Step 1: Select an element that you need to check on the Streaming screen

Step 2: The PROPERTIES box will show the properties of the selected element –> choose a property to create an expectation that you want.

If you want to check the UI of the screen, select the icon camera inside the “Expectation” column to capture screen as reference for future comparison.

If you want to use AI Visual check, select icon to add AI Visual check expectation to compare visual images which help you increase accuracy and reduce effort for testing. AI Visual check will capture the screen to be a visual reference for later test and let you know if during regression test, the actual screen has difference with the captured screen.

You can drag mouse to specify which part is dynamic data and can be excluded in AI Visual Comparison. After that, you click on “OK” button to save Expectation or click on icon to delete selected data area.

You can also click the link referance image in Expectation to view again or edit chosen data. Click on icon if you want to delete AI Visual check expectation.

You can view evidence at Tester 4.0 [Test Result] page.

The system also supports comparing icons of element when confirming expectation. Please follow these steps:
1. Click on element on the streaming screen to select element want to check icon
2. Click on [+] button besides text [Image] in Custom Properties. Then popup Compare Image will be displayed.

There are 2 options: Upload image and use curent

Option 1: Upload image:

Tool allows uploading existing images or images cutted from XD files (JPG, JPEG, PNG format). User can upload image to compare icon by following steps:

Step 2.1. User click on tab Upload image.

Step 2.2. Click and select image or drag file to that area to upload

Step 2.3. Click [OK] button

After selecting image, photo is uploaded to ATOMP storage. If there is an existing image, the new uploaded image will replace the old one. Then, the expectation added will be displayed as < File name of the upload image>

Option 2: Use current image

ATOMP Studio allows capturing automatically the curent screenshot. Then basing on the element position, tool will crop the image of that element automatically. User can use current image to check icon by following step:

Step 2.1. Click on button to use current element’s image as reference.

Step 2.2. Click on [OK] button.

6. Feature IF – Then – Else

Step 1: Add [IF] block

On the [Predefined Actions] tab, click on [Command] menu to expand it.

Expand [Conditional] sub-menu

Click on [IF Statement] to add IF block to script

After that, new block will appear on Recorded Script section.

Hover on to open menu

Click on “Add condition” add a new condition to current [IF] block, it will open input forms as above.

Click on “Add Sub-IF” to add a [Sub-IF] block to current block

Click on “Add ELSE IF” to add [ELSE IF] block after current [IF]block

Click on “Add ELSE” to add [ELSE] block after current [IF] block.

Click on “Delete Block” to delete current block. If deleting a [IF] block will delete all [ELSE IF] and [ELSE] blocks that come with it. User must confirm before action to be executed.

2) Click on to stop working on current block

3) Click on to stop working on current block, to work on a block (pin – unpin block)

4) Delete a condition

5) Edit condition

6) Logical operator (uses when block has 2 more conditions)

7) Element to do comparison operation

8) Property of element (must be specified in order to save condition)

9) Operator

10) Value to compare (default is element’s property value)

11) Click to save condition

12) Click to cancel input condition

13) Block action (will be executed if condition is met)

Some cases when working with IF – THEN – ELSE function
- When adding condition without select element or property to do comparison operation, a message will appear like bellow and condition will not be saved until user specify element & property.
  When deleting all [IF] – [ELSE IF] block condition, a notification will appear with bellow content to inform user that block condition is empty
- When working on [IF] – [ELSE IF] block which does not contain any condition, script will not be recorded and a notification with bellow content will appear
- When trying to save script with an empty condition [IF] – [ELSE IF] block, script will not be saved and notify an error message like bellow

Preview Script with IF – THEN – ELSE function
- All [IF], [ELSE IF], [ELSE] block which condition(s) does not meet will be ignore & show error message like above
- When previewing script with empty condition [IF] – [ELSE IF] block, script will break at block which condition is empty!

6. Android Head Unit

To quickly open functions such as settings, maps and navigation, users can perform the following steps:

Step 1: On the script creation screen, click on the “Device” item from the right menu.

Step 2: Select the “Android Head Unit” item then select “Launch Settings”, “Launch Maps” or Launch Navigation”. Script will be generated on the script screen.

7. Feature Select Step to take evidence on Tester4.0

Step 1: Tick on checkbox in the last column of test script which step you want to take evidence when running on Tester4.0

8. Preview Script

Step 1: Click the [Stop] button to finish recording the test script then click [Preview script] button

-> After selecting device, click the button to start previewing the script.

9. Save a script

Step 1: Click on the [Save script] button -> A dialog will be displayed

Step 2: Input the reference ID and description for the script

Step 3: Click on the [Save Test Script] button

➩ After a test script is saved, it can be found in the screen “Manage Test script” in Tester 4.0

10. Edit a script

Users can edit the created script by following these steps:

Step 1: From the “Manage Test Script” screen, select the “Edit” icon from the script you want to edit => Select device screen will be displayed => Perform a device selection just like creating a normal script

Step 2: After the device is selected => The script creation screen will be displayed, and the users can edit the script normally => Then the users can save the script by 2 methods:

Save to the existing script
Save as a new script

IV. Mirror 1 action on multiple devices

Step 1: Click on the icon mirror -> The pop up list of available devices will be displayed.

Step 2: Select a device that will mirror the actions of the main device.

After selecting a mirror device, the streaming of all mirror devices will be shown.

User’s actions in the main device will also be performed in the mirror devices.

Step 3: Capture the evidence on all devices by clicking on the icon capture screenshot from the tab bar.

èScreenshots in all devices will be shown (both main device and mirror devices)

Step 4: Click on the icon under each image => Screenshot will be copied to the clipboard.

Step 5: Click on the [Download] button

Part 2: Execute Script with Tester 4.0

Tester 4.0: Acts as a Runner or Scheduler to execute the test scripts on many devices.

I. Execute a Script and check the test result

1. Run Now

Step 1: Click on the button [Run Test Script Now] -> The popup Run Test Script will be displayed

Step 2: Select Device (In the Run Test Script popup, select one or more Android devices)-> Click on the [Next] button

Step 3: Select Test Script (After selecting Android devices, the popup list of Android test scripts will be displayed –> select one, or more scripts that the users want to run –> Then click on the [Next] button)

Step 4: Select test language(s)

Input a Job name and select test language(s)
Click on the button [Run Test Now]

Step 5: View running test script(s) (Show the correct number of devices and the name of each device selected for running from step 2 -> Click on the [View] button next to each device name to show the streaming screen of that device)

Step 6: After the test script execution is finished: The Test Report will be shown in the Project Detail screen. To view the detail of a Test report: click on each test report item in the list

2. Run a script On Schedule

Step 1: Click on the [Add Schedule Job] button

➩ The popup select devices & test scripts (similar to the “Run Now” feature) will be displayed

Step 2: Setting the time and date to run test scripts automatically:

Input the Job name
Select test language(s)
Select a time to run test scripts: including the time and the day of the week
Click on the [Add Schedule Test] button

➩ After finish adding a Schedule, at the scheduled time, the selected test scripts will be executed and the test result will be saved.

3. Check the test result

After the test script execution is finished, the Test Report will be shown in the Project Detail screen.

To view the detail of a Test report, please click on each test report item in the list.

User wants to review detail report of test script, user clicks on that test script. Detail report will be displayed.

There are 3 types of results: Pass, Fail and Warning.

Pass: Script works, no script errors, test script return result exactly as expected.

Fail: There was a script error or an input error, not getting results fromTester 4.0. Or there is no script error but Tester 4.0 returns resultdifferent from expectation. If the result is fail, the system will displaydetail of error and error step.

If user use AI to check the expectation in test script, it will have more result: Warning. The script works, there are no script errors, the AI returns the results, but the AI’s result is that something is wrong. Specifically: An element’s image comparison function returns FALSE. When the result is warning, the user needs to confirm the result.

AI when comparing 2 screen shots will give the same ratio of 2 images, the value runs from 0 to 100%. Tester 4.0 defines a threshold level, if it is below this level, it is considered a problem. For example, if threshold setting is 95%:

– A.I returns 2 images that have similarity greater than or equal to 95% -> no problem. Tester 4.0 will return the result – Pass.

– A.I returns 2 images with the same 95% or less -> considered “problematic”. Tester 4.0 will return the result – Warning.

At the moment, the threshold setting is 95%.

When the result is Warning, user needs confirmation by follow steps:

Step1: Clicking on icon then dropdown list will be shown.

Step 2: If user wants to confirm OK, please click on [Confirm OK] button. Popup Note for this confirmation will be displayed. User inputs the reason then clicks on [Confirm] button. The result of test script will change from warning to pass.

Step 3: If user wants to confirm NG, please click on [Confirm NG] button. Popup Note for this confirmation will be displayed. User inputs the reason then clicks on [Confirm] button. The result of test script will change from Warning to Failed.

4. Quick review

After the test script execution is finished, the Test Report will be shown in the Project Detail screen.

To view the detail of a Test report, please click on each test report item in the list. The report of job will be shown as picture.

If the user wants to review the test results quickly, please click on [Detail view] button.

After selecting detail view mode, test scripts with failed or warning results will show error steps and steps using AI visual check.

For test scripts with warning results, user can confirm the results directly by clicking on the Confirm OK/ NG button on the quick review screen.

To view the image larger, please [View Large] button to view it.

User can click on [List view] button to review test report to see the report in the original list form.

Part 3: Remote Control device with Device Farm

Device Farm: Is the Web portal that helps users to remote control a large number of devices of many types & models.

Log in to Device Farm: https://devicefarm.atomp.io/

I. Set up Executable files for the project

After users log in, in the Device List screen, users can see all the devices that are permitted to use.

Click on any device whose status is “Use” (which means that the device is available to use).

II. Interact with device

Interact with device

Users can use computer mouse to click, swipe, tap and hold on the device screen to interact with device

2. Hard keys

In Hard Keys item, when user clicks on the hard keys on Device Farm, they also press the hard buttons on the real device. When user clicks on any hard key, the corresponding screen will be displayed.

3. Restart worker.

When the device is laggy, difficult to manipulate or cannot be manipulated, the user can click the Restart worker button to disconnect the device from the system and reconnect.

After restarting worker, Popup Device was disconnected will be displayed. User can click [Try to connect] button to reconnect or click [Go to Device List] button to select device.

III. Sound

Users can listen to the sound played from the device by clicking on the “Sound” icon.

IV. Capture Screenshot

Users can interact with the device via the Streaming screen and can also capture the screenshot of the device

Step 1: Select the tab [Screenshots]

Step 2: Move to the screen needed to capture the screenshot

Step 3: Click on the button [Screenshot]

V. Device information

Users can view the device information by selecting the “Info” item

Users can view the device’s files by selecting the “File Explorer” item

When the developers and testers want to collect the logs of the device they are using, they can go to “Logs”. And they can choose the type of logs they want to collect by selecting the “Logcat Level” item

Users can perform operations on the device such as opening the camera, turning the volume up or down and rebooting the device in the most convenient way by selecting the “Advanced” items.

ATOMP.IO

Automation Test

Device Farm

Test Management