Androids dreaming of stormy weather

[cdecker]

There have been a couple of attempts to get c-lightning running on Android devices, and I thought it might be a good idea to pull experiences together to see what can be done to make Android support a possibility and make the process of getting it running on Android simpler.

I know that the abcore team (@greenaddress and @lvaccaro) has done some preliminary work to get c-lightning to compile on Android (greenaddress/bitcoin_ndk#21). The approach they followed is:

• Patch the makefile, the endiannes library and the JSON-RPC
• Generate the config.vars and gen_header_versions.h on device, since the configurator runs on the compiler host
• Finally compile the dependencies and lightningd using clang for the target arch

Ideally we'd backport the patches they needed to apply in order to internalize that part. @NickeZ on the other hand has been tackling the need to pre-generate the target config.vars and gen_header_versions.h files in #3464. That'd allow us to compile the configurator for the target arch and run it using the QEMU wrapper. Hopefully that eliminates the need for pre-canned "generated files".

Finally @remyers is also working on getting Android support working, with a host configurator but otherwise cross-compiling for ARM: #3473

Ideally we'd pool our collective experiences in getting the cross-compiled flow working, and document it for others to find in future.

What do you guys think? Are there any stumbling stones that I am missing? Could this allow us to cross-compile reproducible builds as well?

[Saibato]

Nice, i.e. i did once run c/ln compiled in termux on the phone from old source source that i patched step by step to reach master side by side with a full node over orbot tor proxy, stopped this effort a year or so.
The pruned full node i synced fast on a laptop and then insert the sd card in the phone.
Then i did some patches on the core node and c/ln to not fall behind and get block info even on prunded node, i.e. if you know funding unspent that's not perfect but better than nothing.
Since the power consumption is high and the old phone had a bad battery pack i stopped this and lets be honest i am sure the custodial's apps incl. BlockS. efforts would not react welcome like, if i begin to crush there BM on there own platform before moon, so i stopped work on this.
We should use there VC bugs improve ln and crush them later.
Hopefully others have time and can improve, and can in general reduce power consumption of c/ln !!!

[darosior]

What do you guys think?

I think that since I've started working on moving our Bitcoin backend to plugins, I've started dreaming of having C-lightning + a Bitcoin plugin hitting my esplora API englobed in a Kivy Android app 😁.
So I'm very much interested in this !

[cdecker]

Hopefully others have time and can improve, and can in general reduce power consumption of c/ln !!!

Is it really c-lightning's power consumption or is it the co-located bitcoind? In my experience c-lightning is rather lightweight, running on a shoe-string budget.

[Saibato]

Is` it really c-lightning's power consumption or is it the co-located bitcoind?

Sure more bitcoind! And c-lightning is rather lightweight, but can we ever see soverain trustless private pseudonymous LN and make up for the sins of NAT, without a full node?

And will it not be in the near future, that LN is integral part of a matching core node?

In general SW dev in my view should not only in sight of global warming have an eye on power consumption, every byte touched or moved is one byte to much.
Even if some magic wizard comes up with an LN idea to have cold sign keys and offline nodes that just need a short sync to push or forward tx. There will always be big hubs or merch. that run there nodes 24/7. And i guess there will be many scenarios, where LN on there phone will be there only POS. 😅

̶E̶D̶I̶T̶:̶ ̶S̶h̶o̶r̶t̶ ̶s̶i̶d̶e̶ ̶n̶o̶t̶e̶,̶ ̶c̶o̶l̶o̶s̶s̶u̶s̶ ̶h̶t̶t̶p̶s̶:̶/̶/̶t̶w̶i̶t̶t̶e̶r̶.̶c̶o̶m̶/̶s̶c̶i̶e̶n̶c̶e̶m̶u̶s̶e̶u̶m̶/̶s̶t̶a̶t̶u̶s̶/̶1̶2̶2̶5̶0̶2̶1̶4̶8̶7̶1̶2̶1̶3̶3̶8̶3̶7̶0̶ ̶w̶a̶s̶ ̶a̶l̶l̶e̶g̶e̶d̶l̶y̶ ̶u̶s̶e̶d̶ ̶t̶o̶d̶a̶y̶ ̶i̶n̶ ̶4̶4̶ ̶f̶o̶r̶ ̶t̶h̶e̶ ̶1̶s̶t̶ ̶t̶i̶m̶e̶,̶ ̶w̶e̶ ̶s̶h̶o̶u̶l̶d̶ ̶h̶a̶v̶e̶ ̶a̶l̶s̶o̶ ̶a̶n̶ ̶o̶p̶e̶n̶ ̶e̶y̶e̶ ̶o̶n̶ ̶N̶u̶m̶b̶e̶r̶W̶a̶n̶g̶,̶ ̶s̶o̶ ̶s̶o̶m̶e̶t̶i̶m̶e̶s̶ ̶m̶o̶r̶e̶ ̶m̶a̶g̶i̶c̶ ̶b̶y̶t̶e̶s̶ ̶a̶r̶e̶ ̶g̶o̶o̶d̶.̶

[remyers]

I'm going to close #3473 because I can now build and run clightning on my android phone using the bitcoin_ndk project referenced in greenaddress/bitcoin_ndk#21.

My setup: MacOS Catalina 10.15.2 (Macbook Pro, 8GB RAM, 128GB HD)

1. Follow steps to install docker-machine & virtualbox (MacOS guide)

 $ brew install docker docker-machine
 $ brew cask install virtualbox
  	Look for: “installer: The install was successful.”

Note: For MacOS you need to create symlinks to sha256sum so the final report at step 7 works, but it does not prevent creation of the executable artifacts. (MacOS guide)

2. [optional] Set the location where you want the docker-machine files and certificates to be stored, if you don't have enough disk space at the default location.

$ export MACHINE_STORAGE_PATH=/Volumes/Personal/android/machine
$ export DOCKER_CERT_PATH=/Volumes/Personal/android/machine/certs

3. Clone the bitcond_ndk project:

 $ git clone https://github.com/greenaddress/bitcoin_ndk.git
 $ cd bitcoin_ndk
 $ git checkout cln_test

4. Create the docker-machine image using virtualbox driver and call it ‘default’.
   Note: I allocated 100 GB for my disk, but 20 GB is probably sufficient. I also allocated 3 GB of RAM but more is better, especially if you use num_jobs=4 instead of making the changes I made in step 6.
   Note: $PWD should point to the folder you cloned bitcoin_ndk into.

 $ docker-machine create --driver virtualbox --virtualbox-disk-size "100000" --virtualbox-memory "3072" --virtualbox-share-folder /Users/Users --virtualbox-share-folder $PWD:/repo default
	Look for: “Docker is up and running!”
 $ docker-machine env default
 $ eval "$(docker-machine env default)"

5. Follow the steps from the bitcoin_ndk/README.md to build the bitcoin_ndk docker image:
   $ docker build .

6. I made some small changes to bitcoin_ndk so it would build faster on my resource limited computer:
   fetchbuild.sh:53 I used 'num_jobs=2' instead of 4
   run.sh:8: I used 'ARCHS="aarch64-linux-android=64" ' instead of building for every architecture
   run.sh:17,18: I commented out the bitcoinknots and elementsproject 'build repo ...' lines

7. The final step in bitcoin_ndk/README.md will use the docker container to build the artifacts for each target machine.
   $ ./run.sh

8. Unzip the tar files created for your target device and copy them to your android phone:
   Note: The c-lightning executables cross-compiled for my aarch64-linux-android Pixel 2 XL are only about 36MB on disk.

 $ tar -xf aarch64-linux-android_bitcoin.tar.xz 
 $ abd push aarch64-linux-android_bitcoin/* /data/local/tmp
 $ adb shell
  taimen $ cd /data/local/tmp
  taimen $ ./lightningd --version

You should expect to get back: "v0.7.3-modded"
You are using the wrong target build if you get back: "/system/bin/sh: ./lightningd: not executable: 64-bit ELF file"

I did some preliminary testing with a lightning config file on my phone which accesses a regtest bitcoind on my host machine. The bitcoin.conf on my host machine allows RPC access to connections on my local network:

# $HOME/Library/Application Support/Bitcoin/bitcoin.conf
txindex=1
regtest=1
rest=1
rpcallowip=192.168.1.0/255.255.255.0
rpcauth=foo:8526a5a90d3d7d53967dfb1920e1ade0$d88b4cc7f35abdcccd4db591626f861c2f408b28a52c483bd9e5c3827efdfb66).

I tested it like this:

$ abd shell
taimen $ cd /data/local/tmp
taimen $ mkdir regtest
taimen $ mkdir regtest/plugins
taimen $ cp plugins/* regtest/plugins
taimen $ cat << EOF > regtest/config
# file:/data/local/tmp/regtest/config
network=regtest
# daemon
log-level=debug
log-file=/data/local/tmp/regtest/log
rescan=5
bitcoin-cli=/data/local/tmp/bitcoin-cli
bitcoin-datadir=/data/local/tmp
bitcoin-rpcuser=foo
bitcoin-rpcpassword=bar
bitcoin-rpcconnect=192.168.1.20
EOF
taimen $ ./lightningd --lightning-dir=/data/local/tmp/regtest &
taimen $ ./lightning-cli --lightning-dir=/data/local/tmp/regtest listfunds
{
   "outputs": [],
   "channels": []
}

Many thanks to @lvaccaro, the bitcoin-ndk team and everyone else who has helped make cross-compiling clightning for android possible, and now even easier to do.

[lvaccaro]

Thanks for sharing @cdecker :-)
@remyers there are some more fixes on bitcoin_ndk due to better support cross-compilation steps on different machine. You could find full binaries at https://github.com/lvaccaro/bitcoin_ndk/releases/tag/v0.18.1.2

I really like add a CI for cross-compilation (github or gitlab) but this requires a bit more effort to run configurator as just explain. I'll try to keep in touch :-)

[remyers]

@remyers there are some more fixes on bitcoin_ndk due to better support cross-compilation steps on different machine. You could find full binaries at

Great, I will sync the latest changes. It is really helpful to have your build scripts so I can build with some small modifications to c-lightning we are using to test making channel updates over mesh radio.