Cloud Device Testing

Table of Contents

• Introduction
• BrowserStack
  • Using BrowserStack with Gondola
• NTT Remote TestKit
  • Using NTT Remote TestKit with Gondola
• AWS Device Farm
  • Using AWS Device Farm with Gondola
• Sauce Labs
  • Real Device Cloud Testing with Sauce Labs and Gondola
  • Emulator/Simulator Cloud Testing with Sauce Labs and Gondola

Introduction

Many times you'll want to run tests on many different real devices running several OS versions. This could require purchasing quite a few phones. Luckily, real device testing is available in the cloud. Here we'll describe how to use several cloud device testing providers with Gondola.

BrowserStack

Prerequisites:

• Register with BrowserStack

Using BrowserStack with Gondola

• Login to BrowserStack

• Go to Product > App automates.

• In Quick Start Guide, upload your App.

• Next add the information to your gondola.json file.

  • app will be your appurl from above
  • browserstack.user and browserstack.key are your username and access key and are located on the BrowserStack App Automate page.
  • browserstack.appium_version is the Appium version you'd like to use for testing

{
    "output": "./results",
    "helpers": {
        "Appium": {
            "require": "./node_modules/@logigear/gondola/built/helper/AppiumEx",
            "hostname": "hub-cloud.browserstack.com",
            "port": 80,
            "app": "bs://<yourBrowserStackAppURL>",
            "desiredCapabilities": {
                "device": "Samsung Galaxy S8",
                "browserstack.user": "<yourBrowserStackUserName>",
                "browserstack.key": "<yourBrowserStackKey",
                "browserstack.debug": true,
                "browserstack.appium_version": "1.14.0"
            }
        },
        "GondolaHelper": {
            "require": "./node_modules/@logigear/gondola/built/helper/gondolahelper"
        }
    },
    "plugins": {
        "screenshotOnFail": {
            "enabled": "true"
        },
        "allure": {}
    },
    "tests": "./built/tests/loginValidation.js",
    "name": "SampleAndroid"
}

NTT Remote TestKit

Prerequisites:

• Register with NTT Remote TestKit

Using NTT Remote TestKit with Gondola

1. Download & install NTT Remote Testkit

2. Start NTT Remote Testkit & login.

3. Rent a device & enable ADB connection (Android) or enable USB connection (iOS). In case you want to test multi-devices, you can rent more than one. With iOS devices, please make sure you run NTT Remote Testkit on macOS machine which already has XCode installed.

   

   

4. With Android devices, run adb devices to check if your device is connected. With iOS, check your device's connection status in XCode. :::TIP Note When deploying an .ipa file on an iOS device, please make sure the device you're renting has been added to your Apple provisioning profile :::

5. Configure gondola.json to match the device.

AWS Device Farm

Prerequisites:

• Register with AWS Device Farm

Using AWS Device Farm with Gondola

1. Create an IAM User with full AWS Device Farm permissions

   

2. Login and create new project, then create a new run and choose your application

   

3. Upload "<your app>".apk (for Android) or "<your app>".ipa (for iOS) to AWS Device Farm. With iOS test, please build the ipa for a real device (not a simulator)

4. Create a Gondola ABT project. Make sure to run these commands

   • npm i
   • npm run compile

5. Put all of files and folders (except src folder) into a zip file and upload, then choose type execute by "Appium Node.js"

   

6. Edit the .yml file, here are some examples (please replace your target versions of Appium server, Android and iOS):

version: 0.1

        # Phases are collection of commands that get executed on Device Farm.
        phases:
        # The install phase includes commands that install dependencies that your tests use.
        # Default dependencies for testing frameworks supported on Device Farm are already installed.
        install:
            commands:
            # The default Appium server version is 1.7.2.
            # You can switch to an alternate supported version from 1.6.5 and above by using a the avm command, for example "avm 1.7.1"
            # OR
            # To install a newer version of Appium use the following commands:
            - export APPIUM_VERSION=1.15.1
            - avm $APPIUM_VERSION
            - ln -s /usr/local/avm/versions/$APPIUM_VERSION/node_modules/.bin/appium  /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/bin/appium.js
            
            # The default node version is 11.4.0
            # you can switch to an alternate node version using below command.
            # - nvm install 10.13.0
            
            # Unpackage and install the node modules that you uploaded in the test phase.
            - echo "Navigate to test package directory"
            - cd $DEVICEFARM_TEST_PACKAGE_PATH
            
            
        # The pre-test phase includes commands that setup your test environment.
        pre_test:
            commands:
            # We recommend starting the appium server process in the background and using the command below.
            # The Appium server log will go to $DEVICEFARM_LOG_DIR directory.
            # The environment variables below will be auto-populated during run time.
            - echo "Start appium server"
            - >-
                appium --log-timestamp --device-name $DEVICEFARM_DEVICE_NAME
                --platform-name $DEVICEFARM_DEVICE_PLATFORM_NAME --app $DEVICEFARM_APP_PATH
                --udid $DEVICEFARM_DEVICE_UDID --chromedriver-executable $DEVICEFARM_CHROMEDRIVER_EXECUTABLE  >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 &
            
            - >-
                start_appium_timeout=0;
                while [ true ];
                do
                    if [ $start_appium_timeout -gt 60 ];
                    then
                        echo "appium server never started in 60 seconds. Exiting";
                        exit 1;
                    fi;
                    grep -i "Appium REST http interface listener started on 0.0.0.0:4723" $DEVICEFARM_LOG_DIR/appiumlog.txt >> /dev/null 2>&1;
                    if [ $? -eq 0 ];
                    then
                        echo "Appium REST http interface listener started on 0.0.0.0:4723";
                        break;
                    else
                        echo "Waiting for appium server to start. Sleeping for 1 second";
                        sleep 1;
                        start_appium_timeout=$((start_appium_timeout+1));
                    fi;
                done; 
            
        # The test phase includes commands that run your test suite execution.
        test:
            commands:
            # Go into the root folder containing your source code and node_modules
            - echo "Navigate to test source code"
            # Change the directory to node_modules folder as it has your test code and the dependency node modules.
            - cd $DEVICEFARM_TEST_PACKAGE_PATH
            
            - echo "Start Appium Node test"
            # Enter the command below to start the tests . The command should be similar to what you use to run the tests locally.
            # For e.g. assuming you run your tests locally using command "node YOUR_TEST_FILENAME.js.", enter the same command below:
            - >-
                export CAP="{\"output\":\"$DEVICEFARM_LOG_DIR/Results\", \"helpers\": {\"Appium\": {\"app\": \"$DEVICEFARM_APP_PATH\",\"desiredCapabilities\": {\"appActivity\": \"com.logigear.SplashActivity\", \"platformName\": \"$DEVICEFARM_DEVICE_PLATFORM_NAME\",\"platformVersion\": \"9.0\",\"deviceName\": \"$DEVICEFARM_DEVICE_NAME\"}}}}"
            - >-
            - echo $CAP
            - >- 
                ./node_modules/.bin/gondola -c gondola.android.json run --steps --plugins allure --licenseServer trial.gondolatest.com:5555 --override "{\"output\":\"$DEVICEFARM_LOG_DIR/Results\", \"helpers\": {\"Appium\": {\"app\": \"$DEVICEFARM_APP_PATH\",\"desiredCapabilities\": {\"appActivity\": \"com.logigear.SplashActivity\",\"platformName\": \"$DEVICEFARM_DEVICE_PLATFORM_NAME\",\"platformVersion\": \"9.0\",\"deviceName\": \"$DEVICEFARM_DEVICE_NAME\"}}}}"
            - >-

        # The post test phase includes are commands that are run after your tests are executed.
        post_test:
            commands:

        # The artifacts phase lets you specify the location where your tests logs, device logs will be stored.
        # And also lets you specify the location of your test logs and artifacts that you want collected by Device Farm.
        # These logs and artifacts will be available through the ListArtifacts API in Device Farm.
        artifacts:
        # By default, Device Farm will collect artifacts from following directories
        - $DEVICEFARM_LOG_DIR

 version: 0.1

        # Phases are collection of commands that get executed on Device Farm.
        phases:
        # The install phase includes commands that install dependencies that your tests use.
        # Default dependencies for testing frameworks supported on Device Farm are already installed.
        install:
            commands:
            # By default, this testspec file uses Appium server version 1.9.1 which is a prerequisite for iOS 12
            # You can switch to an alternate supported version from 1.6.5 and above by using a the avm command, for example "avm 1.7.1"
           
            - export APPIUM_VERSION=1.9.1
            - avm $APPIUM_VERSION
            - ln -s /usr/local/avm/versions/$APPIUM_VERSION/node_modules/.bin/appium  /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/bin/appium.js
            
            # The default node version is 11.4.0
            # you can switch to an alternate node version using below command.
            # - nvm install 10.13.0
            
            # Unpackage and install the node modules that you uploaded in the test phase.
            - echo "Navigate to test package directory"
            - cd $DEVICEFARM_TEST_PACKAGE_PATH
            - npm install *.tgz
            
            
        # The pre-test phase includes commands that setup your test environment.
        pre_test:
            commands:
            # We recommend starting the appium server process in the background using the command below.
            # Appium server log will go to $DEVICEFARM_LOG_DIR directory.
            # The environment variables below will be auto-populated during run time.
            - echo "Start appium server"
            # The default WDA used is at DEVICEFARM_WDA_DERIVED_DATA_PATH_V1 (Supports versions iOS 12 and below), it is using commit f865d3. See (https://github.com/appium/appium-xcuitest-driver/tree/f865d32e78a5a8a15469bee30ed2f985d378575d)
            # If you need an older WDA version or need support for node modules, use the WDA at DEVICEFARM_WDA_DERIVED_DATA_PATH_V0. (This version does not support iOS 12)
            - >-
                appium --log-timestamp --device-name $DEVICEFARM_DEVICE_NAME
                --platform-name $DEVICEFARM_DEVICE_PLATFORM_NAME --app $DEVICEFARM_APP_PATH
                --udid $DEVICEFARM_DEVICE_UDID --automation-name XCUITest
                --default-capabilities "{\"usePrebuiltWDA\": true, \"derivedDataPath\":\"$DEVICEFARM_WDA_DERIVED_DATA_PATH_V1\"}"
                >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 &
                
                
            - >-
                start_appium_timeout=0;
                while [ true ];
                do
                    if [ $start_appium_timeout -gt 60 ];
                    then
                        echo "appium server never started in 60 seconds. Exiting";
                        exit 1;
                    fi;
                    grep -i "Appium REST http interface listener started on 0.0.0.0:4723" $DEVICEFARM_LOG_DIR/appiumlog.txt >> /dev/null 2>&1;
                    if [ $? -eq 0 ];
                    then
                        echo "Appium REST http interface listener started on 0.0.0.0:4723";
                        break;
                    else
                        echo "Waiting for appium server to start. Sleeping for 1 second";
                        sleep 1;
                        start_appium_timeout=$((start_appium_timeout+1));
                    fi;
                done; 

        # The test phase includes commands that run your test suite execution.
        test:
            commands:
            # Go into the root folder containing your source code and node_modules
            - echo "Navigate to test source code"
            # Change the directory to node_modules folder as it has your test code and the dependency node modules.
            - cd $DEVICEFARM_TEST_PACKAGE_PATH
            
            - echo "Start Appium Node test"
            # Enter the command below to start the tests . The command should be similar to what you use to run the tests locally.
            # Assuming you run your tests locally using command "node YOUR_TEST_FILENAME.js.", enter the same command below:
            - >-
                export CAP="{\"output\":\"$DEVICEFARM_LOG_DIR/Results\", \"helpers\": {\"Appium\": {\"app\": \"$DEVICEFARM_APP_PATH\",\"desiredCapabilities\": {\"appActivity\": \"com.logigear.SplashActivity\", \"platformName\": \"$DEVICEFARM_DEVICE_PLATFORM_NAME\",\"platformVersion\": \"9.3\",\"deviceName\": \"$DEVICEFARM_DEVICE_NAME\"}}}}"
            - >-
            - echo $CAP
            - >- 
                ./node_modules/.bin/gondola -c gondola.ios.json run --steps --plugins allure --licenseServer trial.gondolatest.com:5555 --override "{\"output\":\"$DEVICEFARM_LOG_DIR/Results\", \"helpers\": {\"Appium\": {\"app\": \"$DEVICEFARM_APP_PATH\",\"desiredCapabilities\": {\"appActivity\": \"com.logigear.SplashActivity\",\"platformName\": \"$DEVICEFARM_DEVICE_PLATFORM_NAME\",\"platformVersion\": \"9.3\",\"deviceName\": \"$DEVICEFARM_DEVICE_NAME\"}}}}"
            - >-

        # The post test phase includes are commands that are run after your tests are executed.
        post_test:
            commands:

        # The artifacts phase lets you specify the location where your tests logs, device logs will be stored.
        # And also let you specify the location of your test logs and artifacts which you want to be collected by Device Farm.
        # These logs and artifacts will be available through ListArtifacts API in Device Farm.
        artifacts:
        # By default, Device Farm will collect your artifacts from following directories
        - $DEVICEFARM_LOG_DIR

7. Select from one of the available device pools or create a new device pool

   

8. Review and run the test

9. Open the test summary for each device and download the "Customer Artifacts" to get your test results.

Sauce Labs

Prerequisites:

• Register with Sauce Labs and sign in

Real Device Cloud Testing with Sauce Labs and Gondola

1. Click the Access Real Device Automation link in the top right
   
2. Create a new app by pressing the New App button
   
3. You can upload your app directly to SauceLab's servers or host it remotely
   
4. After uploading your app you will be take to the device's management page.
5. Go to basic setup page for the uploaded app to get required capabilities for your gondola.json configuration (“AUTOMATED TESTING”>>Appium>>Setup Instructions).

Real Device gondola.json samples

{
  "output": "./results",
  "helpers": {
    "Appium": {
      "protocol": "https",
      "host": "us1-manual.app.testobject.com",
      "port": 443,
      "path": "/wd/hub",
      "app": "sauce-storage:<YourApk>.apk",
      "require": "./node_modules/@logigear/gondola/built/helper/AppiumEx",
      "smartWait": 5000,
      "desiredCapabilities": {
        "testobject_api_key": "<YourTestObjectApiKey>", // This is unique for each Test Project
        "platformName": "Android",
        "platformVersion": "8.0.0",
        "automationName": "UiAutomator2",
        "newCommandTimeout": 600,
        "name": "Mobile App Test"
      }
    },
    "GondolaHelper": {
      "require": "./node_modules/@logigear/gondola/built/helper/gondolahelper"
    }
  },
  "plugins": {
    "screenshotOnFail": {
      "enabled": "true"
    },
    "allure": {}
  },
  "tests": "./built/tests/*.js",
  "name": "SampleAndroid"
}

{
  "output": "./results",
  "helpers": {
    "Appium": {
      "protocol": "https",
      "host": "us1-manual.app.testobject.com",
      "port": 443,
      "path": "/wd/hub",
      "app": "sauce-storage:<YourIpa>.ipa",
      "require": "./node_modules/@logigear/gondola/built/helper/AppiumEx",
      "smartWait": 5000,
      "desiredCapabilities": {
        "testobject_api_key": "<YourTestObjectApiKey>", // This is unique for each Test Project
        "platformName": "iOS",
        "platformVersion": "12.1.4",
        "automationName": "XCUITest",
        "connectHardwareKeyboard": false,
        "newCommandTimeout": 600
      }
    },
    "GondolaHelper": {
      "require": "./node_modules/@logigear/gondola/built/helper/gondolahelper"
    }
  },
  "plugins": {
    "screenshotOnFail": {
      "enabled": "true"
    },
    "allure": {}
  },
  "tests": "./built/tests/*.js",
  "name": "SampleiOS"
}

Note

• app is the location to user’s uploaded app file(apk/ipa/zip). There are many way to upload app file. Please read this article
• Users can choose between US and EU device servers. The information on each server is shown under “Setup Instructions”.
• testobject_api_key is a unique ID for each Test Project. It is shown under “Setup Instructions” and required for running tests.
• platformName and platformVersion are required. There is a list of supported device with version details under “Setup Instructions”. (Click on the View Devices button)
• More information about capabilities for running test on Real Device is shown here.

Emulator/Simulator Cloud Testing with Sauce Labs and Gondola

Navigate to https://app.saucelabs.com/dashboard then go to “User Settings” to get your Driver Creation URL

The Driver Creation URL’s format is: https://$username:$Access_KEY@ondemand.saucelabs.com:443/wd/hub An example URL for an account with “abc” as username and “1234-5678” as access key: https://abc:1234-5678@ondemand.saucelabs.com:443/wd/hub

{
  "output": "./results",
  "helpers": {
    "Appium": {
      "protocol": "https",
      "host": "<YourUsername>:<AccessKey>@ondemand.saucelabs.com",
      "port": 443,
      "path": "/wd/hub",
      "app": "sauce-storage:<YourApk>.apk",
      "require": "./node_modules/@logigear/gondola/built/helper/AppiumEx",
      "desiredCapabilities": {
        "platformName": "Android",
        "deviceName": "Android Emulator",
        "appiumVersion": "",
        "deviceOrientation": "portrait",
        "browserName": "",
        "platformVersion": "8.0"
      }
    },
    "GondolaHelper": {
      "require": "./node_modules/@logigear/gondola/built/helper/gondolahelper"
    }
  },
  "plugins": {
    "screenshotOnFail": {
      "enabled": "true"
    },
    "allure": {}
  },
  "tests": "./built/tests/loginValidation.js",
  "name": "SampleAndroid"
}

{
  "output": "./results",
  "helpers": {
    "Appium": {
      "protocol": "https",
      "host": "<YourUsername>:<AccessKey>@ondemand.saucelabs.com",
      "port": 443,
      "path": "/wd/hub",
      "app": "sauce-storage:<YourIpa>.ipa",
      "require": "./node_modules/@logigear/gondola/built/helper/AppiumEx",
      "smartWait": 5000,
      "desiredCapabilities": {
        "appiumVersion": "",
        "deviceName": "iPhone X Simulator",
        "platformVersion": "13.0",
        "platformName": "iOS",
        "browserName": ""
      }
    },
    "GondolaHelper": {
      "require": "./node_modules/@logigear/gondola/built/helper/gondolahelper"
    }
  },
  "plugins": {
    "screenshotOnFail": {
      "enabled": "true"
    },
    "allure": {}
  },
  "tests": "./built/tests/loginValidation.js",
  "name": "SampleiOS"
}

Note

• app is required and specifies the link to the app.
• Use the Driver Creation URL to configure the host property (required)
• More information about capabilities for running test on Emulator/Simulator is available here
• Use the Platform Configurator to get the list of supported Emulator/Simulator devices and correct values for desired capabilities.
• For more information on uploading your app, click here