bpo-28833: Fix cross-compilation of third-party extension modules by xdegaye · Pull Request #17420 · python/cpython (original) (raw)

Since I was flagged on this one, I figured I'd address @vstinner's queries from perspective of the BeeWare project.

I can confirm that we (i.e., BeeWare) see this manifestations of this problem on both Android and Apple platforms (iOS most notably, but notionally on tvOS, watchOS as well).

Supporting mobile platforms essentially faces four problems:

  1. Building Python for the platform architecture (or architectures)
  2. Building binary modules for the platform architecture (or architectures)
  3. Installing modules for a platform other that the development platform.
  4. Using those modules at runtime, on the target device.

At present, (1) is supported reasonably well (although some external patching and tooling is required), but (2) and (3) are not supported due to some baked in assumptions about how Python will be invoked and used (in particular, that the host platform and target platform are assumed to always be the same). The mechanisms supporting (4) vary depending on the platform.

BeeWare uses the manufacturer provided developer toolchains (Xcode et al for Apple; Android Studio and Android NDK for Android/WearOS). Historically, we've used VOC to cross compile to Java bytecode on Android. We've put that approach on hold recently; our recent PSF Education grant is based on pursuing an embedded CPython approach. However, that decision doesn't alter many of the cross-platform targeting problems we're facing.

The Xcode and Android Studio toolchains both work based on cross-compilation. The manufacturer-provided tooling does not support running compilers on device. Any source code is compiled on an x86_64 desktop machine, for a target architecture (and usually multiple target architectures).

This means:

  1. CPython itself needs to be compiled for multiple ARM architectures, using x86_64 as the host architecture.
  2. python setup.py needs to able to generate binary modules and wheels for multiple ARM architectures, using x86_64 is the host architecture.
  3. pip install needs to be able to be told to install an iOS/Android wheel, when the host platform is macOS/Linux/Windows.

(1) is supported reasonably well at present (with some patching and external tooling required). The problem comes with (2) and (3). This patch appears to be addressing (2).

From BeeWare's perspective, the issue is building and distributing wheels, not APK packages. APKs are designed to distribute entire, self contained applications, not libraries. If I'm building an application, and I need to use a third party package with a C module, I need to be able to install that module into my application's site_packages. My application will be distributed as an APK, but an APK is not a helpful or useful medium of exchange for a third party Python module.

APK packaging is required to distribute final applications. However, BeeWare has come to the position that setuptools isn't well suited to application packaging; we're in the process of converting Briefcase into being a standalone PEP518 compliant build tool that can be used to produce application installers for distribution through traditional App Store channels.

To date, BeeWare has taken the strategy of avoiding problems of binary modules, only claiming to support pure Python modules. Limited support for binary modules has been achieved by embedding the code for those modules into the application code itself (i.e., embedding the libraries into the binary of the native application). However, this isn't a generalized solution. We'd much rather be able to pip install an Android/iOS wheel from PyPI, or at least cross-compile source to produce the relevant binary modules when building the native application for distribution.

I am aware that some Android Python users/developers are using the device as a development platform (so they have GCC, pip etc hosted on device); however, this use case is not an option for BeeWare. App stores (especially iOS App Store) strictly control the ability of distributed applications to download external dependencies that aren't distributed with the application; on-device development processes violate those requirements.

I'm happy to provide any any further clarifications or details if anyone wants them.