Unable to run unit tests in CI on an Android emulator running on Genymotion SaaS

[taras]

Environment

• CLI: 5.4.0-2019-03-21-13075
• Cross-platform modules: 5.3.0
• Android Runtime: 5.2.1
• iOS Runtime: 5.2.0
• Plugin(s):

Describe the bug

We're using Genymotion SaaS to spin up emulators to run tests on CI. I got it wired up to start the emulators and tns test android is able to install the app on the emulator but tests don't actually run. I get the following output.

You should be able to see the entire log by going to https://circleci.com/gh/resideo-platform/exemplar/3033?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link

Note: the emulators runs on a remote server. I have to forward any port from remote emulator to the local host. There is an error about not being able to connect to a port: Error is: connect ECONNREFUSED 127.0.0.1:43108.. I tried to open a port but it changes every time that the test run. I haven't been able to track down what this port is used for.

To Reproduce

I believe you should be able to reproduce this issue by starting a remote emulator where the testing port needs to be explicitly opened.

Expected behavior

I'm not sure if the tests are failing to run because the port is not open, but if it is then I'd expect the port to be stable. At least, it should be possible to specify the port that's used so I can open that port.

Sample project

The entire project is available in the repo https://github.com/resideo-platform/exemplar/pull/61

Additional context

Thank you :)

[Fatme]

@taras,

Thank you for reporting this issue!

As we can see from the logs, you try to execute the following commands:

yarn test android --available-devices
yarn test android --bundle --justlaunch --device=build-${CIRCLE_BUILD_NUM}

It seems the error is from the first command and the second is not executed at all.

We suspect the problem might be caused by adb. During the device detection process we spawn different adb commands in order to get information for device. Let's try to isolate the issue and see if this is the case. In order to do it, we'd like to play a little bit with the tns devices command. So, let's replace the above commands with the followings:

yarn devices --log trace
yarn devices --available-devices --log trace

The new logs should give us information about the executed adb commands and their output.

We'll be more than glad to help you to investigate the problem!

[taras]

Hi @Fatme ,

Thank you for helping me with this :)

I made your changes and pushed a commit https://github.com/resideo-platform/exemplar/runs/97241299

Here is a running job https://circleci.com/gh/resideo-platform/exemplar/3107?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link

[taras]

Ok, it's rebuilding because I forgot to add tns to each of those commands.

https://github.com/resideo-platform/exemplar/runs/97248705

[taras]

It seems like it's not able to find the device :( https://circleci.com/gh/resideo-platform/exemplar/3136?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link

[Fatme]

Yep, the device is unavailable - The device instance is not connected to adb or is not responding.
Currently we support only devices connected via usb, we don't support remote devices over network.

We'll discuss the problem and come back to you when we have more information.

[cowboyd]

It seems like this is a blocker to being able to run Android tests in a CI environment.

[Fatme]

Hey @taras, @cowboyd

Actually it is possible to use Android tests on CI environment, we have a similar setup here. We understand your case is different
as your emulator is running on a remote server.

We noticed that adb devices outputs the connected devices in the following format - localhost:35037. It's a little bit strange as we're expecting to be something like <IP address>:35037. We suggest you to try the steps here and after that check again the output of adb devices command.

We'll be glad to assist you further in this case as using remote Android emulator is in our future plans, so this initiative might bring value for both sides.

[taras]

Hi @Fatme,

It's a little bit strange as we're expecting to be something like :35037
I updated to use 127.0.0.1 instead of localhost - same result.

We suggest you to try the steps here and after that check again the output of adb devices command.

gmsaas instances adbconnect cfc87e11-369b-436d-8c40-772a273ac126 --adb-serial-port=35037 is equivalent to running adb tcpip 35037. As a result, adb devices -l shows the following

List of devices attached
127.0.0.1:35037      device      product:vbox86p device:vbox86p transport_id:1

devices between the IP and the description means that the device is connected to adb.

Here is the job log https://circleci.com/gh/resideo-platform/exemplar/3276?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link

Let me know if you'd like to jump on a call and try to work through this together.

[Fatme]

@taras, @cowboyd,

Sorry for the late reply!

As we have some critical issues on live and too much tasks for the upcoming 5.4.0 and 6.0.0 releases, we don't have enough time to investigate this problem. We don't forget it and most probably will come back to it next week.

[taras]

@Fatme thank you for staying in communication with us - we really appreciate it.

We're not able to run automated testing for Angular at the moment, but we're actively developing and testing on iOS so it's bearable at moment. We're planning to start working on functional testing at the end of May, hopefully you'll have a chance to look at this issue by then.

Let me know if there is anything we can do to help with this issue.

[Fatme]

Hey @taras, @cowboyd,

We’ve finally reached out to this issue and played a little bit with the execution of unit tests on Genymotion emulator on remote machine. Let me summarize the results.

The first thing we tried was to run unit tests on real device that is not connected to the machine via USB cable. We were really happy as it worked out of the box from the first attempt without any additional actions from our side. The steps are very simple:

• Connect the device to the computer via USB cable and make sure the device is found from adb using adb devices command
• Execute adb tcpip 5555
• Disconnect the device as removing the USB cable
• Go to the Settings —> About phone -> Status to view the IP address of the phone.
• Run adb connect <device’s IP address>:5555
• Run adb devices again
• Run tns test android —bundle and ensure the tests are executed on the device over wifi

The second thing we tried was to start a Genymotion emulator on a remote machine and tried to connect to it. We executed exactly the same steps and as a result - the unit tests were successfully executed on Genymotion emulator. That was our second win.

The next goal was to investigate why unit tests are not executed using gmsaas. Unfortunately, we don't have Genymotion Cloud SaaS account so we're not able to try it. Most probably the problem is in gmsaas instances adbconnect command or maybe some missed configuration/setup. We're not sure what exactly might gone wrong with gmsaas.

We suggest you to eliminate the tns from the picture and try to connect to the Genymotion emulator using only adb commands. We currently know that adb devices command works, but adb shell command doesn't work. The goal is to have working adb shell command.
In other words, remove the followings

yarn tns devices --log trace
yarn tns devices --available-devices --log trace

and replace them with

adb shell getprop

We're expecting to see here device offline error or some similar error.

The idea of trying this is to determine if adb itself is able to connect to the Genymotion cloud emulator in the way required by our CLI (i.e. with adb shell). If this command succeeds, we'll have to investigate why it does not work when it is started from CLI.
If the command fails (i.e. shows error that device is offline, or something else), we suggest you to contact Genymotion support for clarification if adb shell is supported on their side and how to configure the service to be able to execute it.

Can you try the suggested approach and inform us for the result?

[taras]

@Fatme thank you for the thorough investigation. My Genymotion account expired so I emailed Genymotion to refill my trial account. I'll let you know when I'm able to run the command. Again, thank you very much for staying on top of this.

[taras]

@Fatme I got the account re-activated and ran adb shell. It looks like adb gets connected to the device correctly.

You can see the output here https://circleci.com/gh/resideo-platform/exemplar/4976?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link