Using Mergeall on Android with Termux (and Pydroid 3) (original) (raw)

Preface: as of May 2023, the system described on this page has been superseded by the newer _PC-Phone USB Sync_—a standalone backup/sync app that runs on both Android 8-and-later devices, as well as Windows, macOS, and Linux PCs. Read about and fetch the new app at its separatewebsite, and tap the button below to learn how this app improves on prior solutions.

On Android 11 and later, this new app uses the All files access permissionto regain access to both shared storage and removable (e.g., USB) drives. This circumvents the loss of file-path/POSIX access to USB drives in later Androids. Shared storage is still slower in recent Androids, but the new app supports content hosting in faster app-specific storage, and UFS 4.0 chips promise to speed storage in general. As a self-contained app, the new program also eliminates the Termux and Pydroid 3 app dependencies of this and other formersolutions, and provides a much more seamless and graphical Android experience.

On PCs, the new program is delivered as a standalone app or executable to avoid extra installs and mesh with the host platform's paradigms. And on every platform, this app is still an all-Python system—despite Android's Java biases. It embeds and uses the portable Mergeall system to process content; codes its GUI in Python with the portable Kivy toolkit; builds apps and executables with the buildozer and PyInstaller tools; and talks to Java with pyjnius only when it must. In fact, this is still largely Mergeall, with a new and improved GUI launcher that also provides access to USB drives on recent Androids.

Like all the earlier Android/PC content-sync solutions worked out at this site over the last five years, the new app is also private by design: its use of USB drives for folder syncs both boosts speed, and ensures that your content remains your property—not someone else's point of control. This app is available as full and trial versions on the Play store for Android, and as free downloads for PCs. Please visit the new app's websitefor more screenshots, more info, and download links.

This document describes how to use Mergeall to sync content on Android devices. It primarily focuses on running command lines in the Termux app, and more briefly explores a Mergeall GUI alternative on this platform. This coverage is provided freely, but with no warranties of any kind. Findings here reflect testing on Samsung Galaxy devices running Android 7, 8, and 9 (Nougat, Oreo, and Pie) and 10. Given the wide variety of products and rapid—really, supersonic—rate of change in the Android world, applicability to other devices and Androids cannot be guaranteed. Mergeall itself changes content by design: see its usage cautionsbefore using the approaches outlined here.

Contents:

Short story: most devices that run Android 11 and later can no longer use the tools described here, but can use a newer and equivalent alternative.

Both the script and GUI solutions described in this doc require Android Oreo or later, and a device with access to either a USB drive or a microSD card for syncs. Because Android 11 revoked USB access for POSIX programs like Mergeall, users of devices which lack a microSD card and run Android 11 or later should instead use the Android Deltas Sync alternative solution availablehere and describedhere.

This alternative syncs content from PC to phone with delta sets computed from a proxy drive instead of direct comparisons, but works equally well, and requires roughly the same amount of user interaction. For more on its rationale, tap here. For more on Android 11's changes, click here. microSD cards may still be accessible to tools here on Android 11, but are increasingly rare.

Overview

If you're able to meet its requirements below, Mergeall on Android provides incremental backups and changes propagation for content stored on your smartphone. With a few simple commands (or a fledgling GUI), you can sync content on your phone's internal storage or removable SD card both to and from an attached USB drive, and can do so right on your phone itself. The net effect turns your phone into a portable storage device for arbitrarily large content collections, without requiring card removal or device rooting, and without the cost, speed, and privacy negatives of cloud storage.

The packagethat hosts the guide you are reading is a supplement to the Mergeall system. It comes with precoded _helper scripts_that you can run directly on your phone from the Termux app's prompt. Though not required, these scripts are designed to simplify Mergeall's command-lines mode on Android, and minimize the risk of user errors. To make edits on smartphones easier, these scripts also use common settings in the included .bash_profile file, which you'll modify for your drives' IDs and folder names. Besides their role in scripts, these settings provide quick command-line access to your folders in general.

For Mergeall users who break out in hives at the thought of using command lines, this guide's Appendix B also includes coverage of using Mergeall's GUI in the Pydroid 3 app, and this package includes patch files needed to make the GUI work. While this approach comes with code changes, operational issues, and pay-or-else advertising that make it a secondary choice here, some users may prefer a GUI even if it entails extra drama. If you wish to explore this option, be sure to read the Termux coverage in the main part of this guide first, as most of it applies to Pydroid 3 GUIs too.

Android's history and variability make documentation challenging. In this guide and its scripts, the terms "SD" and "USB" are used to mean a removable microSD card, and anything that works on the USB port, respectively, and the terms "external" and "removable" are used interchangeably for a drive that can be detached from your phone (i.e., SD or USB). Your device also has nonremovable storage called "internal" here, though Android often refers to this as "external" and makes it available in a folder named /sdcard. And if you don't have a removable card, your phone instead has a nonremovable "external" partition on your phone's internal drive set aside for storage—and confusion.

In the name of generality (and sanity), this package's scripts use the more function-related "data" to mean the content copy kept on your phone (e.g., on internal storage or removable SD card), and "plug" to refer to the content copy on your attached USB drive (e.g., on flashdrive or portable SSD); extrapolate as needed for your use cases. The term "phone" crops up informally here too, but this guide applies to Android devices of all types.

And to any grammarians in this document's audience: some of the closed compounds used here, including "logfile," "zipfile," and "fullscreen" may rankle a few purists, but they just make sense (the compounds, that is, not the purists).

Mergeall Requirements on Android

The Mergeallsystem, originally developed for PCs, can be used unchanged on Android to synchronize on-phone content to and from a USB drive, without rooting your phone, and without requiring SD card removals—an arguably amazing feat in such a limited form factor. To make this work, though, you'll have to meet a few requirements up front. In short, Android has multiple issues and permission rules that require a custom approach; to use Mergeall on this platform, you:

The sections that follow go into these requirements in more detail. Later sections cover setup for these requirements, and usage of this package's helper scripts. The positive take here is that Mergeall on Android may be constrained, but it does work. While some users may prefer pulling their SD card and syncing on a PC, this is delicate work in the extreme, and isn't possible on all devices. With a USB drive, a little up-front work, and either simple command lines or a GUI, Mergeall brings a more PC-like experience to your on-phone content management.

Use Android 8 (Oreo) and Higher Only

The first requirement is an absolute: Mergeall works on Android devices, but only if they are running 2017's Android 8 (a.k.a. Oreo, and sometimes just O) or later. Earlier Androids have a showstopper bug which renders content-sync tools like Mergeall unusable. You can still use Mergeall to manage your phone's content prior to Oreo, but not on Android itself: remove your card and sync it on your PC.

The short story on the bug is that earlier Androids are unable to copy file modification times to either FAT32 and exFAT drives or internal storage on non-rooted phones: file content can be copied correctly, but copied files are always stamped with the time at which the copy was made, not that of the source. This is lethal to content-sync tools like Mergeall, which rely on file timestamps for change-detection speed. Without timestamp copies, on-phone content copies cannot be compared to other copies accurately, and quick content syncs are impossible.

This issue is well-known, long-lived, device-wide, and not specific to Mergeall, Python, or Termux. In fact, a shell cp -p copy command fails to copy times with an error before Oreo too—see the Nougatlogfile. It stems from Android's former FUSE-based filesystem, which required root access to update file times but not content, and was eventually replaced with Samsung's SDCardFS. For more background, try the bug reporthere, the thread here, and searches like this and this. As an example of its wide swath, this bug also kills calendar-file backup copies in the Frigcal program, which makes a cameo appearance in Appendix B; prior to Oreo, many programs were impossible on smartphones.

Though likely unintentional, it's unfortunate that the world's most-used mobile operatingsystemdidn't support non-cloud content syncing until 2017 (and still doesn't for many prior-version users). The good news is that this bug is now fixed, making cloud-free storage on Android finally practical for non-trivial content—and just in time for the larger-storage devices beginning to appear.

Update: in mid-2020, Mergeall has also been verified to run on Android 10(formerly called Q), using the latest Termux and Pydroid 3 apps and their latest Pythons—specifically: Termux 0.99, Pydroid 3 4.0, and Python 3.8 in both. This means you can run its command lines and GUI on Androids Oreo (8), Pie (9), and 10. Android 11 (formerly R) support is still murky, given its planned change to internal-storage permissions; Mergeall will likely work on 11, but content may have to be nested barring app work-arounds.

Use FAT32 External Drives on Some Devices

Preface: This section documents an exFAT timestamps issue that exists in former Samsung Androids. The issue has been fixed as of 2020's Android 10 per the epilog, but will still be present on older unupdated devices. Hence, users of devices running Android 9 (Pie) and earlier are still advised to use FAT32 per this section, but users of newer or upgraded Samsung devices running Android 10 or higher may freely employ exFAT for external drives hosting Mergeall-managed content—and can safely skip both DST-rollover issues and this section.

The Issue

Due to a current Samsung Android bug regarding modification times recorded by the exFAT filesystem, removable drives used with Mergeall on Android should generally use FAT32 instead of exFAT for interoperability with other platforms. In short, the modification times of files created on exFAT drives are known to be incorrectly skewed as follows when using Android Nougat and Oreo (and later Pie) on multiple Samsung devices:

Both skews were observed in and are relative to the US Pacific time zone, which is normally 8 hours behind the UTC base. These are likely symptoms of the same exFAT bug in Samsung Android, and may eventually be fixed in its future versions, but they can wreak havoc with Mergeall file comparisons today.

It's important to note up front that, due to testing constraints, it's unknown whether this flaw lies only in Samsung software or is more widespread. exFAT is not part of Android's AOSP core-functionality stack (described here and here) which all phone makers augment with code from various sources (described here andhere), and the issue is labeled "Android" in this document for brevity alone. Before making filesystem choices, you'll want to compare exFAT-drive file times on your own PC and Android devices to see if this defect impacts you. Even if limited to Samsung devices, though, this bug's scope is large.

It's also worth noting that you may be able to use exFAT in some contexts anyhow. It's now supported on most Androids; the first bullet above applies only to content created on Mac OS; this may be a Samsung-only issue; and exFAT-drive files may in some use cases squeak by as incorrect-but-good-enough for Mergeall comparisons because their mistakes are consistent. Moreover, users who will only ever backup on-phone content to a USB drive are exempt from exFAT PC interoperability concerns. Until this bug is fixed, though, it's dire enough to recommend FAT32 as the more reliable choice for your Mergeall-on-Android drives.

If you're willing to take that recommendation on faith, feel free to skip ahead to the next requirement now. For the curious and skeptical, though, the following traces the bug's forensics—beginning at the scene of the crime.

Act 1: Mac OS Content Is Skewed +16 Hours

The first symptom of this exFAT bug is easiest to diagnose: in testing on three recent Samsung devices (a J7, an S8+, and multiple Note9s), exFAT file timestamps created on some PC platforms are interpreted differently on Samsung Android than they are on any other platform tested. That is, the same file on the same exFAT drive reports a different modification time on Android alone.

In a comprehensive test conducted in early 2019 in the Pacific time zone:

In other words, this test's results make the flaw appear to be a double bug: the Mac OS PC platform records exFAT timestamps in such a way that triggers an incorrect interpretation on the Android mobile platform. Files created on exFAT drives by Mac OS (only) report incorrect times on Android (only).

No, really. For a more graphic look at the problem, here is a folder of files created on Mac OS High Sierra, as it's seen on Mac OS El Capitan,Windows 10, and Android Oreo("Today" on Mac OS means 1/15/2019, and 1/16 was still tomorrow when this test was run). As you can see, the file times are wrong by 16 hours on Android alone—files created around 8:20 AM report their times near 12:20 AM the next day.

Conversely, files created by this test on Android report the same and correct times on Mac OS,Windows, and Android. That makes this seem a one-way bug: it impacts just exFAT files created on Mac OS and viewed on Android, and not vice versa. You'd also never notice it if you stick to PCs (Windows times are correct onMac OSand vice-versa), or limit your scope to just Windows and Android (Windows times are correct on Android and vice-versa). As the next section will explain, though, the latter deduction falls apart in the face of fresh evidence.

Act 2: File Saves May Be Skewed -16 Hours

But wait, it gets worse. The second symptom of Android exFAT anomaly is subtler and took more digging to unearth: in addition to the prior section's +16-hour Mac OS content skew, Samsung Android devices may also write file modification times incorrectly on exFAT drives. In later tests run on the same Samsung Android devices, files saved to internal storage and FAT32 drives recorded modification times correctly, but some saves to an exFAT USB drive erroneously recorded times that are interpreted as 16 hours behind the actual save times on every platform except Samsung Android itself.

No, really. The facts in this category are bizarre and even appear to be random. The incorrect times saved by Android are always incorrect on both Mac OS and Windows, but not all times saved by Android are incorrect (which is why this effect was uncovered later), and those that are wrong seem to depend on program and/or context (e.g., one program wrote new files' times correctly but existing files' times skewed—sometimes).

This makes this skew difficult to trigger, but easy to spot when it appears. As a brief example, a folder of files on an exFAT USB drive which began like thiswas updated and extended by assorted programs on Android around 11:30 AM on 2/27/2019 (again, in the Pacific time zone), yielding the following scenes on Android, Windows, and Mac OS; as you'll notice, some of the Android-written timestamps are wrong by -16 hours on the latter two.

That is, it's not just reads, and it's not just Mac OS: the whole conception of time in whatever software does exFAT on Samsung Android disagrees badly with others. This is really just the other side of a single bug's coin, and this guide doesn't have time or space to document it in more detail (and it's really not this document's job to do so in any event). But there's clearly something quite wrong with exFAT on Samsung Android today.

The Mergeall Damage

Gory details aside, this is another showstopper for timestamp-based programs like Mergeall: because exFAT files don't process timestamps correctly on Android, such files are generally unusable across any platform mix that includes Android. These files' incorrect times may work on Android itself—Mergeall doesn't care that times are correct, only that they differ when files are changed (i.e., consistently incorrect is good enough). However, the time skews can still cause problems when using exFAT drives in at least three noticeably devious ways:

Just as insidious, there seems no way to compensate for the Android time skews automatically. For example, because exFAT drives may contain content created on multiple platforms, just some of your files, those saved on Mac OS, will report incorrect times on Android, and programs like Mergeall cannot tell which differences are valid and which reflect +16-hour Mac OS skew. The -16-hour write skew will be present on all PC platforms, and its nearly random nature seems even more impossible to work around.

Happily, FAT32 drives both interpret and record times correctly, and are thus immune to this bug. For instance, here's what the same file saved on Mac OS looks like on Mac OS andWindows; on Android, this same file's time is skewed on an exFAT drive, but correct on a FAT32 drive. This means FAT32 provides a path for using Mergeall on Android. Less happily, this precludes mixed FAT32/exFAT usage: because only exFAT drives skew times, their files won't compare correctly with the same files on FAT32 drives (as for others) and spurious differences and recopies will ensue.

The Verdict

In sum, this defect renders exFAT generally unusable on Samsung (and perhaps other) Android devices, and makes it impossible to recommend exFAT for Mergeall on this platform today.

Ultimately, this is a bug in Samsung or Android code that should be fixed. Although file times may have been broken on Android in the past, exFAT files today still report and record incorrect modification times on the Samsung Android platform alone, and neither file sources nor program contexts justify further deferring a repair. Barring a future Samsung or Android fix, though, your safest bet is to use FAT32 for all the removable drives you will sync with Mergeall on Android for both accuracy and interoperability.

This is regrettable, given exFAT's support for larger files and compatibility with other filesystems' times. Per its well-known limitations, FAT32 doesn't support files larger than 4G, and you may need to manually adjust your FAT32 drives' modification times twice a year to keep them in sync with non-FAT drives after daylight-savings-time (DST) rollovers. The internal storage in both your PC and phone, for example, is usually a non-FAT drive whose UTC times clash with FAT's local times.

See the FAT times fixer tool covered aheadfor a simple way to make the adjustment; if you use FAT32, you'll run this tool at DST rollovers on your intermediate USB drive (to match your PC and phone), and possibly on a removable SD card (to match your intermediate drive). You might also run this tool on Samsung Android to adjust the times it skews on exFAT drives, but this will then skew them on all other devices (yes, !), and requires extra steps too; FAT-32 seems a better fix.

The upside is that, thanks to Microsoft patent constraints, FAT32 is still much more widely usable today than exFAT. More importantly, its timestamps are implemented more consistently and interoperably; for Mergeall on Android, this is a killer asset.

After this section was written, the exFAT bug it describes was verified to still exist on Samsung devices running Android Pie (9), in addition to Nougat and Oreo. Despite this longevity, the web is woefully silent about this bug—which is hardly surprising, given that timestamps couldn't be copied on Android at all until 2017's Oreo, which made the platform unusable for an entire class of software. Onethreadlooks at a seemingly unrelated issue, but hints at pathology in Mac OS's exFAT implementation in general.

For its part, the Mergeall development team has dutifully posted this bug on both Samsung and Android websites, as well as in the Samsung+ app (the US equivalent of Samsung Members at the time), but doesn't presently hold much hope for a fix. The Android website has already dismissed the bug as an OEM-specific "won't fix" (with no apparent interest in forwarding it to the market-leadingOEM); there's been no substantive reply from Samsung on either its website or its app in a month and a half; and most support forums in the smartphone domain seem more likely to suggest restarting your phone and counting to 10.

Update: shortly after the preceding paragraph was written, Samsung replied on the Samsung Members app in spring 2019 to acknowledge an issue regarding the valid bit for time-zone offset not being set in their exFAT implementation (background detailshere andhere). This makes exFAT files' times correct when created and used only on a phone, but fails to accommodate creation and usage on other platforms, and explains this section's peculiar empirical findings. The reply also promised to try to improve the implementation, but cautioned that a fix may take some time due to dependencies on the current policy. While it's too early to file this under "happy ending," it now seems reasonable to look forward to a future repair; exFAT drives will be much more useful on Samsung devices when those drives are interoperable with PCs, and the computing world at large.

Update: the Mergeall team is pleased to report that the Samsung exFAT time-skew bug has been fixed as of Android 10. Specifically, a Note 10+ upgraded in 2020 to Samsung's Android 10 reports modtimes on exFAT drives that correctly match those on PCs, and copies them correctly to both internal and removable Android storage; view the screenshots proofhere. This means you won't run into time-skew issues when using an exFAT-formatted drive for Mergeall on Samsung devices running Android 10 and higher, and can avoid FAT32's 4G-filesize limit and DST-rollover adjustments. That said, you'll still want to use FAT32 for drives that may be used on older Samsung Androids—of which only Oreo (8), Pie (9), and 10 can be used for Mergeall at all. Android 11 is a mixed bag; the Samsung exFAT fix is still in place, but this Android seems certain to hobble Mergeall in new and exciting ways...

Use Mergeall Command Lines in the Termux App

On Android, Mergeall's Tk-based GUIworks marginally with caveats, but its command-line modeworks fully and qualifier free. To use this mode, you simply need to use Mergeall's source code, and type command lines to launch it. Although there are multiple ways to run Mergeall's Python source code on Android, the Termux app provides a full-featured Linux experience that seems best suited to the Mergeall task. Neither Termux nor command lines are required for Mergeall on Android, but they're suggested for most users.

In more detail, Mergeall's apps and executables for PCs (Mac OS, Windows, and Linux) do not apply to Android, and its reliance on the Tk library (and Python 3.X's tkinter interface to it) means that using its GUI's source code on this platform entails major tradeoffs. Although there is some new support for using Tk in Python programs run on Android today, it comes with rude advertising, substantial glitches, and extra steps that make it difficult to recommend for Mergeall syncs. See the notebox ahead for more on this story; it's evolving, but at this writing Tk may still be a nonstarter for many Mergeall users on Android.

Luckily, Mergeall is built with a decoupled architecture that allows its main code to be run separately from its GUI. To use Mergeall's command-line mode on your phone, you'll need to:

We'll walk through all these steps in more detail later in this guide. This is obviously more work than installing and opening a standalone app, but it's the easiest workable option for using Mergeall on Android today. On the bright side, this isn't nearly as difficult as it may sound. This package's helper scripts simplify the required command lines; the necessary installs are really just a few minutes of easy up-front work; and Termux provides a complete and general-purpose Linux command-line environment where Mergeall's source code is fully at home. As we'll learn later, a content sync is as simple as entering this at the Termux prompt:

bash mergeall-android-scripts/mergeall-plug-to-data-update.sh

This is roughly just a dozen taps, with the tab completion stressed repeatedly in this guide (and can be even faster if saved to a file). Although Mergeall's GUI comes with major downsides on Android today, its helper-script commands should still be easy enough to appease even the most chronically command-line phobic.

For an alternative approach, check out Appendix B's coverage of running Mergeall's GUI with the new tkinter support in the Pydroid 3 app's IDE. There, you'll learn more about using Python tkinter GUIs on Android, and sample a leading alternative to Termux for running Python code on this platform. Because this alternative comes with significant tradeoffs—and shares most of the requirements and setup tasks covered here for Termux—it's been relegated to the supplemental-reading section of this guide. Despite its rough edges, the GUI does work in Pydroid 3, and you may wind up preferring it (and this author may too); but this guide's Android topics still apply, and its suggestions are for a user base at large.

Grant Termux Extra Storage Permissions

In addition to installing Termux, you must run its utility scripttermux-setup-storage to create Termux's writeable app-specific folders, and allow it to update files in internal storage and access removable drives more broadly. See this pagefor more on the Termux setup command. Without it, Termux—and the Python scripts like Mergeall that it runs—cannot read or update most of the folders on your phone's internal storage or removable drives. This makes content-processing programs like Mergeall unusable in Termux until you run this command.

The Before Permissions Picture

To illustrate the Termux permissions story, let's run some Python code within the Termux app. To do this yourself you'd need to install Termux and Python per ahead, but there's no reason to do so yet; this is just a demo. This also assumes some familiarity with pathnames (jump ahead to this section's notebox if you need more background); assumes basic Python knowledge (but you'll probably get the gist of this simple code either way); and uses Termux 0.65 (if your mileage varies in the future, it's probably due to a new Termux, a new Android, or both).

Before the Termux setup command is run, your storage options are severely limited: on non-rooted phones, you can read and update files in the fully private Termux app folder /data/data/com.termux (the one holding your ~ home), but not much more. In Termux's Python, with internal storage at /sdcard, a removable drive named /storage/8BB9-1202, and tracebacks trimmed for space here:

open('/data/data/com.termux/xxx.txt', 'w').write('1234') 4 open('/sdcard/xxx.txt', 'w').write('1234') PermissionError: [Errno 13] Permission denied: '/sdcard/xxx.txt'

open('/storage/8BB9-1202/xxx.txt', 'w').write('1234') PermissionError: [Errno 13] Permission denied: '/storage/8BB9-1202/xxx.txt'

open('/storage/8BB9-1202/Android/data/com.termux/xxx.txt', 'w').write('1234') FileNotFoundError: [Errno 2] No such file or directory: '/storage/8BB9-1202/Android/data/com.termux/xxx.txt'

This code shows most writes (i.e., updates) failing, but read accesses at all these locations work the same way at this point. As you can see, there's not much accessible before the Termux setup command is run.

The Termux app-private folder at /data is the standout exception. It supports changes out of the box, but is largely useless for Mergeall, because this folder cannot be accessed outside the Termux app on non-rooted devices. This includes file explorers, photo viewers, and text editor apps, and disqualifies this folder for most Mergeall roles. Because you really don't want to store your content in a location that can be accessed by just one app (and a command-line-only one at that), Termux's scope is very narrow when first installed. (But see the update ahead for exceptions.)

Technically, at this point Termux can also read and update its nested Android/data/com.termux app-specific folders like the one in the last command above, provided you make them manually on either removable drives or internal storage first. But you would have to do so in another app or on a PC: because Termux cannot access the containing folders yet, it cannot build the required folder paths under them. To do more, we have to move on.

Update: Termux eventually added support for the Storage Access Framework (SAF), which allows specially equipped file-explorer apps to access the Termux app-private home folder from outside the Termux app. This isn't normal access and subverts Android permissions, but it does allow broader use of this otherwise private folder. For more details, see the later coverage in Android Deltas Sync.

The After Permissions Picture

After running the Termux setup command, your storage prospects improve radically. You—and hence Python scripts like Mergeall—have update access to all of internal storage, plus Termux's nested app-specific folders on removable drives:

open('/data/data/com.termux/xxx.txt', 'w').write('1234') 4 open('/sdcard/xxx.txt', 'w').write('1234') 4 open('/storage/8BB9-1202/xxx.txt', 'w').write('1234') PermissionError: [Errno 13] Permission denied: '/storage/8BB9-1202/xxx.txt'

open('/storage/8BB9-1202/Android/data/com.termux/xxx.txt', 'w').write('1234') 4

Read access for these folders is essentially the same as the write access shown, except that the failing folder here can be read without error too. That is, after augmenting its permissions with termux-setup-storage, Termux, and the programs like Mergeall that it runs:

As noted, Termux's setup command creates its app-specific folders, like the one used in the last command run above, but you can create these folders manually too (and we will in the setup part of this guide). Running the setup command creates these nested app folders if they don't already exist; this ensures that Termux, and the scripts it runs, have a place to make changes on removable drives.

More Permissions Examples

For a more comprehensive look at accessible paths which we'll skip in this guide, see:

By now, you may be feeling a bit confused by folder pathnames if you haven't done much with command lines in the past. For a gentler introduction to the subject, see this guide's Appendix A for extra background that we'll omit here. Like much technical documentation, this guide has to walk a thin line between alienating newcomers, and boring experts.

Update: though uncertain at this writing, Android 11 may impose the same access constraints on internal storage that KitKat imposed on removable media, and invalidate some of this section's coverage in the process. See Appendix C for more on the possible consequences for Mergeall—and every other program that saves shared data on the Android platform.

Nest External-Drive Content in Termux App Folders

After you run the Termux permissions command of the prior section, Mergeall is able to both read and change content anywhere in internal storage, and can read content anywhere on an external (i.e., removable) drive. To allow Mergeall to make _changes_on external drives, though, your content must be nested in the drives' Termux app-specific folders. In Mergeall-speak, this nesting is not necessary if an external drive will only ever be the from copy in synchronizations, but it is required if the drive will ever play the role of to.

In more-concrete terms, the Mergeall content folders that you will only read on your phone can be located anywhere, but those you will change (i.e., update) must reside at either of the following Android locations:

You can use the first of these for on-phone content copies if you have enough space, but must use the second for content on SD cards and USB drives that you wish to update. Since this is an arguably bizarre constraint, let's take a closer look at its rules.

Content in Internal Storage

First, the simple case: if you're lucky enough to have a phone with enough free space to store your content, internal storage is the easiest approach for on-phone storage, and recommended over SD cards. Internal storage:

There's more on internal storage's speed for Mergeall-specific tasks ahead. Internal storage is also growing larger in recent phones, and is the only option for on-phone storage if your device doesn't have a removable card. To store your Mergeall content in internal storage, simply copy it to a folder like the following on your phone; /sdcard really means internal storage, and the MY-STUFF folder name here can be any name you wish:

/sdcard/MY-STUFF

This seems simple—and even PC-like—but internal storage isn't enough by itself. For one thing, even if your on-phone content is in internal storage, you'll still need to sync to and/or from an external USB drive in most use cases. For another, if your phone's available internal storage space isn't large enough for your content (and many today are not), you'll need to store your on-phone content on your device's SD card instead. Since both USB drives and SD cards require nested storage to support updates, let's turn to this model next.

Content on Removable Drives

Removable storage is generally slower, but it's required if your phone's internal space is too small, and is still the ultimate in content control. To store your changeable Mergeall content on external (i.e., removable) storage—including USB drives and your phone's removable SD card—simply copy it to a folder nested in the drive's Termux app-specific folder like this; again, MY-STUFF means your stuff's folder:

/storage/25C9-1405/Android/data/com.termux/MY-STUFF

Android uses IDs like this path's 25C9-1405 to uniquely identify external drives. Your drives' IDs will naturally vary too, and are displayed in popular Android file explorers (including this).More unusual here: on both SD cards and USB drives, content folders like this path's MY-STUFF must be nested six-levels deep in the drives' Termux app-specific folder as shown, if you wish to update them with Mergeall on your phone. This differs from internal storage, where anything under the /sdcard root works.

Termux's explanation for why this is so is told on pageshere andhere. This restriction on updating removable drives came online in Android KitKat—to the dismay of both users and developers—and has seen multiple later mutations, including app-initiated run-time requests for storage permissions. Termux, and all other Python apps tested, however, remain limited to their own app-specific folders for updates to SD cards and USB drives.

This may seem an odd requirement given that internal storage is wide open to changes, but it cannot be lifted in the context of Termux command lines today, and isn't as grievous in practice as it may sound. For instance, you can still access your content normally in a file explorer (as captured in this SD-card screen shot), and a home-screen shortcut (like this one) provides quick access to your nested content folder; see your file explorer to set one up.

To Nest or Not to Nest

Keep in mind that no nesting is required for content stored in your phone's internal storage (i.e., in /sdcard), because it's freely writeable. Moreover, you can store content at the root of USB drives and SD cards too, but only if you'll only ever _read_—and never change—it with Mergeall. Unless you're sure that a removable drive's copy won't be the subject of a future Mergeall update, it should be nested in the drive's Termux folder.

Also keep in mind that Android's permission rules have varied much and often in the past, and seem likely to do so again in the future. While some media may grow more accessible and some less, it's impossible to foresee tradeoffs in a platform that seems to change the rules every other Tuesday (more on this in the following notebox).

This section noted that internal storage is recommended for storing content on your phone because it is more accessible to other apps, but this merits a callout here. Due to the permission rules we met earlier, some apps that do not request broader access, including Termux, are unable to update content on removable media except in their own app-specific folders. This is why you must oddly nest your updateable content on removable media, including SD cards and USB drives, to manage it with Mergeall in this app.

Unfortunately, this can also make it impossible to update such content in some other apps. As we'll see in Appendix B, the alternative Pydroid 3 GUI app has the same access limits—on removable media, it can update only in its own app-specific folder. For such apps, this is a bit of a catch-22: you must nest content on removable media to update it, but this cuts it off from other apps' updates. More tangibly, Pydroid 3 cannot update any content nested in Termux's app-specific folder on removable media—and vice versa. The net effect is that your choice of updating app is binding, unless you move your content.

Luckily, this caveat's scope is limited: it impacts only updates on removable drives, and does not apply to all apps. You can still freely view nested content on these drives, and modify it in general-purpose apps like file explorers and text editors that request broader permissions than Termux and Pydroid 3. If it's important to also update content in apps with narrower permissions, though, _internal storage_is your best option. Especially with the next section's peril factored in, the strange access limitations of removable drives on Android makes them less well-suited for single-folder content storage. Watch for a rehash of this topic when we consider Pydroid 3 at the end of this guide; it becomes crucial for alternative apps.

Update: though uncertain at this writing, Android 11 may impose the same access constraints on internal storage that KitKat imposed on removable media, and invalidate some of this section's coverage in the process. See Appendix C for more on the possible consequences for Mergeall—and every other program that saves shared data on the Android platform.

Retain Your Nested Content if Termux is Uninstalled

Finally, a bold-font caution: per normal Android operation (described by Android docshere and here), uninstalling Termux also silently deletes its app-specific folders on all your drives—including any content folders you've nested there. Unnested content copies, and unnested working folders in internal storage are immune to this, but those nested on your SD card and USB drives to support Mergeall updates may perish with Termux. Because this occurs without clear warning, it's substantially worse than the prior section's caveat on removable-media accessibility, and further bolsters the case for internal storage.

Specifically: on all devices tested, the Android/data/com.termuxfolders on both the removable SD card (at /storage/xxxx-xxxx) and internal storage (at /sdcard) were automatically deleted when Termux was uninstalled, along with any and all content placed there by a user. On one device, a file explorer also attempted to intervene and delete this content too, but the system apparently beat it to the punch. There is no reason to nest your content in an app-specific folder on internal storage—and a root-level folder on internal storage avoids this issue altogether—but there is no other option for updateable content on removable drives in Termux (or other Python apps we'll meet in Appendix B).

This automatic deletion may sound scary, but it's designed to clear associated space when you're done using an app, and really isn't much more dangerous than file-explorer deletions if expected (and an rm command in the Termux shell can be deadlier still). Before uninstalling Termux, though, be careful to move your nested content if you wish to retain it. In Android, uninstalls imply deletions.

And that's a wrap on the requirements front. In exchange for all this, Mergeall provides a free and proven incremental backup and changes propagation tool for the content you store on Android devices. Though not for everyone, this Mergeall approach requires neither device rooting nor card extraction—both of which can be perilous, if not impossible—and brings robust cloud-free storage management to your phone. If those benefits offset this section's requirements for your usage—or you just enjoy a technical challenge in general—read on for more on using this package's helper scripts to simplify Mergeall runs on Android.

If a standalone Mergeall app ever appears, it may obviate the need for both Termux command lines and content nesting in Termux's app-specific folders, but would still incur Android version and filesystem constraints. In particular, barring an exFAT fix, drives would still be best formatted as FAT32 for interoperability. Given the sloth with which bugs are fixed in commercial products like Samsung Android, using FAT32 today is likely not a short-term measure. If you're already using FAT32 drives, the only impact of an app upgrade would be an optional move of content to unnested drive roots.

Update: a year later, that possible future has already changed—an exFAT timestamps fix did eventually appear in Samsung Android 10, which makes FAT32 formatting unnecessary on devices running this or later versions. Android 11's filesystem constraints, however, may pose fresh obstacles for content management, and its latest updates in 2020 may make a standalone Mergeall app much more crucial.

Setting up Your Phone and Drives

Now that you've seen the what's required to use Mergeall on Android in general, this section shifts focus to the how. It provides detailed coverage of steps to follow to prepare your phone and drives for Mergeall use. Here's a quick rundown of these up-front tasks:

  1. Install Termux—to run Mergeall commands
  2. Run termux-setup-storage—to access content
  3. Install Python—to run Mergeall's source code
  4. Make your working folder—to run code and view results
  5. Install this package and Mergeall—to sync your content
  6. Prepare your drives—to add content and meet specs
  7. Edit your bash profile—to set paths used by scripts Most of these are one-time tasks and can be performed in a variety of ways, but they should generally be done in the order listed here. The section following this one covers running merges, but assumes the tasks here have been completed. So grab your phone, PC, and drives, and get ready for a bit of admin work.

1. Install Termux

To be able to run Mergeall's Python scripts, you first need a command-line environment in which to run them. The Termux app provides a featured-rich Linux shell for Android that fits the bill; see its homepage for background. To get started, install the Termux app on your phone from theGoogle Play Store. Mergeall's command-line mode "just works" in Termux, because Android is really Linux under the hood.

Shortly after this guide was penned, Termux changed its extra-keys row to be configurable, but in the process also removed keys crucial to command-line editing (e.g., arrows). To restore the former keys shown in this guide's Termux screenshots, visit this page, and scroll to its "Extra Keys Row" section. In brief: in Termux itself, run a Unix mkdir command to create the new folder~/.termux; use a Unix editor like vi or nano to make a file named termux.properties in the new folder and paste in the line shown on that page; and restart Termux.

Option: some readers may prefer to disable the Termux row entirely and install an alternative on-screen keyboard that has all the important bits, like Hacker's Keyboard.Bonus: such a keyboard may also come in handy for tkinter GUIs run on Android (stay tuned for the details in Appendix B).

2. Run termux-setup-storage

To allow Termux to update both internal storage and its app-specific folders on removable drives, launch the Termux app you just installed on your phone, and run the following command at its prompt (the $ in command listings here is Termux's default prompt; type the text that follows it, and press whatever means "enter" on your keyboard to submit the command):

$ termux-setup-storage

After that, allow access in the popup that appears. Note that you may need to rerun this command later with removable drives connected, if you get permission errors. This command also creates the required /Android/data/com.termux nested folders on all connected media automatically if they do not exist, but you can also make these manually when you're preparing your drives on a PC (as we will a few steps ahead).

If you've forgotten why this permissions step is required (or came here from a search), see the earlier coverage. Also note that Termux commands that update your phone's internal storage from here on in this guide assume that this command has been run; without it, such commands will fail on permission errors.

3. Install Python

Mergeall is coded in the Python programming language. As mentioned earlier, we need to use its source code on Android because its PC executables don't apply, and source code requires a Python interpreter on any platform. To install Python for use in Termux, simply type the following command at the Termux app's prompt on your phone, and answer "y" (yes) when prompted:

$ pkg install python

This gets the latest Python 3.X (version 3.7 at this writing), which works best for Mergeall. If you want to verify the install, type just python at the Termux prompt:

$ python Python 3.7.1 (default, Oct 21 2018, 18:20:26) [Clang 7.0.2 (https://android.googlesource.com/toolchain/clang 003100370607242d on linux Type "help", "copyright", "credits" or "license" for more information.

import sys, os, platform sys.platform, os.name, platform.machine() ('linux', 'posix', 'aarch64')

As usual, a Ctrl+d key combination at Python's >>>prompt gets you back to Termux's $ prompt.

4. Make Your Working Folder

Now that you've got Termux and Python ready to go, it's time to setup a working folder on your phone where you'll run scripts to sync content. This is a simple task, but you'll want to decide up front where you wish to work, to avoid having to repeat steps later.

First, the recommendation: an /sdcard/work folder located in the root of your internal-storage space is an ideal candidate for your working folder, because it's fast, accessible to file explorers and most other apps, changeable by programs run in Termux, and short to type. This folder has the following multiple aliases on devices used in testing, but shorter is generally better on a smartphone:

/sdcard/work /storage/emulated/0/work

You can work in any writeable folder on your phone, but others come with tradeoffs that make them less than ideal in this role. For example:

For this guide, we'll assume and use the recommended working folder in internal storage (and use its shorter /sdcard name), because this is the simplest and most-functional location. To create your folder, run the following in Termux on your phone:

$ mkdir /sdcard/work

You can also create this folder in your favorite Android file explorer—one of the benefits of locating it in internal storage this way. Once you've got your folder set to go, you're ready to populate it with software, per the next step on our list.

5. Install This Package and Mergeall

Wherever you choose to work, unzip (i.e., extract) both this helper-scripts package and Mergeall's source code there for convenience. The helper scripts simply augment Mergeall, so you'll need both packages. You can use the standard Mergeall source-code package, because its core syncing code developed for Mac OS, Windows, and Linux works unchanged on Android—a testament to the benefits of portable software (though the story for Mergeall's GUI in Appendix Bwon't be nearly as seamless).

For readers newer to Unix systems, the following is one way to install this helper-scripts package on your phone by running commands in Termux itself. First, in Termux, go to the working folder you created in the prior step:

$ cd /sdcard/work

Next, fetch (i.e., download) this helper-script package and unzip it to your new working folder; click the links here on your phone to perform the fetch part of this recipe, then type the commands in Termux:

Fetch the zipfile at learning-python.com/mergeall-android-scripts.zip

$ mv ../Download/mergeall-android-scripts.zip . $ unzip -d . mergeall-android-scripts.zip

Finally, do similar to install Mergeall's source-code package in the same working folder:

Fetch the zipfile at learning-python.com/mergeall-products/Mergeall-source.zip

$ mv ../Download/Mergeall-source.zip . $ unzip -d . Mergeall-source.zip

For an example of the resulting working folder, see this screen capture. This assumes that clicking the links above on your phone makes the zipfiles show up in /sdcard/Download, but that's normal Android public-folder behavior.

Two tips here. First, if you're new to Unix, be sure to use Termux's arrow keys to recall prior commands to edit and resubmit, and its tab completion to let Termux fill in most of these commands for you. The latter makes command lines much easier to type; for instance, a tab after "m" in the sessions above likely expands it to "mergeall-android-scripts.zip," one after "M" fills in the Mergeall full zipfile's name, and "Dow"+tab expands to "Download" similarly. Don't type what you don't have to type—especially on a phone.

Second, there are other ways to achieve most of the above that we'll skip here. For instance, you can fetch files on a PC and beam them to your phone; can use Android file explorers instead of command lines for most steps in this process (the former often call an unzip an extract, and refer to . as the current path); and can even download the zipfiles directly in Termux usingcurl after a pkg install curl:

$ cd /sdcard/work $ curl -O https://learning-python.com/mergeall-products/Mergeall-source.zip $ unzip -d . Mergeall-source.zip

Termux has the wget download command too. Regardless of their sizes, Linux-based devices offer an embarrassment of options.

6. Prepare Your Drives

Preface: As noted earlier, the requirement to format removable drives as FAT32 has been lifted as of Samsung Android 10, but still applies to drives used on older versions of that platform. Given that Android updates are not all mandatory, this yields unavoidable bifurcation; this section's parenthetical updates address the schism.

Depending on the size of your content, the next step may take the longest. To use your content on Android, its storage structure must satisfy the media's filesystem and location requirements we met earlier here and here. In brief:

Since this is probably the most convoluted of the steps in this list (and is also required of the GUI alternative in Appendix B), the following sections provide a more detailed breakdown by media type.

To Setup Removable SD Cards and USB Drives

Use your PC to setup the removable drives you'll use on Android. First, you'll need to meet the FAT32 requirement (till Samsung Android 10). If your drive is already using FAT32, you can move on to the next paragraph. Otherwise, start by copying any existing Android content on your drive to a temporary location so it can be restored later; this includes existing DCIM (camera) and Android (app-data) folders, if present. Then, format your drive as FAT32, and restore the prior existing content you saved. If you're new to formatting, see the notebox ahead for tips.

Next, place your content-copy folder on your drive. As noted earlier, if you're sure you will never sync content to a removable drive on Android, this folder can be stored anywhere on the drive, including its root; copy or move it there using your usual tools. To allow Mergeall to update the drive's content, though, its content-copy folder must be placed in Termux's nested app-specific folder, as follows:

For the last bullet above, you can move your content folder to the Termux app folder on the same drive with a single command on Unix-like PCs; here's an example on Mac OS (on Linux, use your media's paths; on Windows use drive letters instead of volume names, and move with DOS commands):

~$ mv /Volumes/EXT256-1/MY-STUFF /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF

A caution here: you can move existing content in a file explorer too, but be sure a move in yours is a true move, and not a copy-and-delete. The speed difference can be staggering for large folders, and some Android file explorers may miss the mark on this.

Whether your content was copied or moved, it's now located in the Termux app folder—and four levels down from the drive's root—which can be inconvenient when coding paths and browsing for files. To make this easier to access on your PC, add a link (or shortcut) to it at your drive's root; here's how to do so on Mac OS with a symlink (that looks like this):

~$ ln -s /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF /Volumes/EXT256-1/MY-STUFF

The link (or shortcut) is optional, but can save a lot of navigation time in file explorers, and typing in command lines. The root-level link probably won't work on Android, and you can't create one for removable drives on non-rooted phones because you don't have the required permissions on removable drives (this is the whole reason for nesting content in the first place); but it can save steps on PCs.

On Android, instead make a home-screen shortcut (that looks like this) to your content folder; a longpress in your file explorer is the usual first step. This makes more sense for a resident SD card than a transient USB drive; but because your content is now six levels deep when viewed on Android—nested in the Android/data/com.termux folder of your drives' /storage/xxxx-xxxx root—the shortcut can save a whopping five taps per access, plus a file-explorer open.

To Setup Internal Storage on Your Phone

If you're able (or required) to store your on-phone content in internal storage, you don't need to reformat as FAT32 (even before Samsung Android 10), or nest your content in the Termux app-specific folder. In fact, you shouldn't—internal storage is already formatted, and nested Termux app folders disappear on Termux uninstalls as noted earlier. Instead, simply create a folder for your content at the root of internal storage, and copy your content there. Both these steps must be done on your phone itself.

To begin, create your content folder in an Android file explorer, or use a Unix mkdir command in Termux itself. For example, the following creates a content folder named MY-STUFF using a command in Termux; as usual, your folder name may vary:

$ mkdir /sdcard/MY-STUFF

Then, copy your content to your phone. For most users, this will employ an intermediary USB drive: copy from PC to USB drive, then from USB drive to phone. A content copy on a USB drive you've already setup for Android in the prior section will suffice, and avoids another step.

A caution here too: on the phone, you can run the copy to internal storage in most Android file explorers, but some fail to copy over file modification times. At least one popular app, for instance, simply stamps every copied file with the current time, which is a showstopper for Mergeall—and no better than the pre-Oreo Android bug (in fact, this may be a lingering accommodation of it).

To make sure times make the move too, use Mergeall's cpall.py bulk-copy utility on your phone instead. There's more on this utility ahead; in short, the copy on Termux looks like the following, assuming the settings we will create in the next and final step are in place. That is, you'll need to return here and run this after setting drive IDs in your bash profile file (USB drive IDs are not available until that drive is prepped and attached, and there's no way to show you this command without them):

$ cd $work $ py Mergeall-source/cpall.py plugplug plugdata -skipcruft -vv -u > Admin-Mergeall/importlog.txt

Here too, once the content copy finishes, you may wish to create a home-screen shortcut to its new location on your phone. In this mode it's only two levels down in internal storage when viewed on Android—at /sdcard/MY-SYUFF—but the shortcut still saves one tap per access, plus a file-explorer open. A symlink doesn't make sense here, because it would be two levels down too (and may be impossible in any event: a ln -s at Termux's prompt didn't work for /sdcard, despite having write permissions there).

A reminder: even if you use internal storage for on-phone content, you'll also need to setup a USB drive to propagate changes between your PC and phone (or save on-phone changes in general). That removable drive will still require FAT32 formatting for interoperability (till Samsung Android 10), and content nesting for updates as described earlier (here and here), and can be used to initially populate your internal storage with content. Your on-phone content, though, is substantially easier to setup and use.

As noted, you'll generally need to reformat your removable drives unless they already use FAT32 (or are used only on Samsung Android 10 or later), and this isn't a trivial task if you're new to it. On Windows, the standard file Explorer won't format larger drives as FAT32 (it forces exFAT instead), but a DOS formatcommand and third-party options will; for pointers, try this search andthis thread.

On Mac OS, the Disc Utility app's Erase happily formats larger drives as FAT32 (if you select this and use uppercase drive names), and Linux users have many options. Android itself can format drives too (see your drive's Format in Settings ⇨ Device maintenance/care ⇨ Storage ⇨ Storage settings); unfortunately, it seems to silently choose FAT32 or exFAT for you based on drive size (a case of user-friendliness gone bad) so you'll have to use a PC for larger drives.

On all platforms, the usual warning about formatting applies: it erases your drive, so copy off anything valuable first. You may also want to add a .nomedia file at the top of reformatted removable drives that will be duplicates of the content kept on your phone (e.g., on a go-between drive that will be used to propagate changes between your phone and PC); where it works, this "hidden" file prevents Android's media scanner from searching the redundant copies.

7. Edit Your Bash Profile

Last but not least, you'll now install and edit the bash profile file shipped in this package. This must happen last, because you may not know the Android names of your drives until they've been prepared and attached to your phone. Formatting, if run, changes the IDs Android uses to identify drives, and you'll need these IDs in this step.

Installing the Profile File

Along with its scripts (and this guide), this helper-scripts package includes a prototype bash profile file named your--.bash_profile. This file has simple settings that provide quick access to your Termux home and working folders, as well as your content-copy directories on your phone and removable drives. You can view this file's contents here; it's mostly documentation (the lines that start with a #).

To use this file, delete the "your--" in its name; edit its path settings as both its comments and the next section describe; and either save this file in your Termux home folder or copy-paste its code into a ~/.bash_profile you already use. Because your Termux home folder is app private, you will normally need to install the profile file there using commands in Termux itself (not a file explorer). Here's the required Unix incantation; this assumes you're working in /sdcard/work and have already unzipped this package there:

$ cd ~ $ cp /sdcard/work/mergeall-android-scripts/your--.bash_profile . $ mv your--.bash_profile .bash_profile

For example usage of this file's settings at the Termux prompt, see this Android screen capture(and zoom in and zoom often). The bash profile file is also integral to using the helper scripts here: per the next and final setup section, it defines Android paths in a single location shared by all.

Editing Your Path Settings

To make both commands and edits easier on smartphones, the helper scripts in this package use common content-copy path variables set in your bash profile file. This way, paths need be changed in that file only, and only when you first install this package or wish to use a new drive.

When you do need to change paths, the required edits are simple: in the ~/.bash_profile you created from the prototype file, the settings to change are marked as "EDIT ME." Replace these precoded paths' "xxxx-xxxx" IDs and literal "MY-STUFF" with your own drive IDs and content-folder name, respectively. Again, your drive IDs can be found in Android file explorers like this (make sure to match the case of drive ID letters exactly), and folder name is whatever you wish it to be. Here's one way to edit the profile file in Termux—because it's not visible to other apps, you'll generally need to edit it in Termux itself too:

$ cd ~ $ vi .bash_profile $ source .bash_profile /sdcard/work

This runs source to load your new settings after you're done, and uses the Unix vi text editor. If vi is too cryptic for your tastes, the more user-friendly and WYSIWYG nanotext editor also works in Termux after a pkg install nano. You can also rename and edit the bash profile file on your PC and beam it over, but you'll still need to copy it into your home folder in Termux, because that folder is visible nowhere else.

Special cases: if you opt to use internal storage for your on-phone content copy instead of SD card (per the requirements section earlier), use the profile file's /sdcard/MY-STUFF data-path setting instead. You may need to also edit the scripts themselves to configure for your Mergeall source-code package's install location if it's not in .(the current working directory), but this is not required if Mergeall's package is unzipped to the same folder as this scripts package.

Because removable drives are identified by drive IDs in Android, the paths used by helper scripts will change if a new removable drive is inserted. To reconfigure for a new drive, simply update paths in your ~/.bash_profile with the editing commands above; all Android helper scripts automatically source (i.e., reload) this file on startup, so they always use its current settings. You may also need to rerun Termux's termux-setup-storage when changing to a new drive; if you get permission errors, this applies to you.

The symlinks made at drive roots by the Mac OS move-and-link script here might suffice for pathnames in other scripts, but these links won't generally work on Android, and non-rooted phones don't have permission to make such links in removable drives on Android itself. Termux's ~/storage/external-* links also record drive paths, but they seem too ambiguous to use: they give IDs but not drive type, and are not always current. Instead, the helper scripts rely on the manual $data and $plugsettings in your bash profile file, both for generality, and because explicit is usually better than implicit—especially when your content is at stake.

Managing Your Content

After you've performed the prior section's setup requirements, you're finally good to go: whenever you wish to sync and compare your content, simply run the helper scripts and other command lines described in the sections that follow. There's generally no reason to repeat setup steps (unless you wish to use a new drive), but the commands here can be rerun anytime you wish to propagate changes to or from your phone.

Running Syncs and Diffs

The helper scripts shipped in this document's folder make it easier to use Mergeall on smartphones, by precoding common use cases, and utilizing shared path settings. Among the collection, you'll find helper scripts that run:

Additional scripts run modification time (a.k.a. modtime) adjustment tools for FAT32 drives, and other Mergeall utilities are available as standard Unix command lines. As a bonus, all the Android helper scripts here automatically save logfile output to your working folder's Admin-Mergeall subfolder, using filenames either passed or generated; to see how this works, let's jump right into the available commands.

Syncing Commands

To sync or compare your content: select the Mergeall helper scriptin this package whose name matches your goal; open the Termux app on your phone; and run the selected script with a bash _scriptname_.sh command line at the Termux prompt. For example, the following launches a USB-to-phone updates run, assuming this package has been unzipped in your working folder:

$ bash mergeall-android-scripts/mergeall-plug-to-data-update.sh

The bash command just tells Termux to run all the precoded commands in the chosen file, and scripts' filenames imply their action. For instance, plug-to-data in this script's name syncs changes from USB drive to on-phone storage, and update means apply changes to the destination automatically; data-to-plug syncs the other way, and report means report differences only without changing anything.

Worth reiterating: be sure to use shell tab completion to fill in the folder and script names in these commands; in this case "bash m+tab/m+tab" is most of the command. Optional: scripts located in your Termux home folder (~) can also be made more directly executable with a chmod +x _scriptname_.shand run with just a _scriptname_.sh; in Termux this works for home-folder scripts only, not for working folders located elsewhere.

Command Variations

Mergeall's scripts—and hence this package's bash helper scripts that run them—both process files and generate logfiles that detail their actions and results. On Android, script logfile output is automatically saved in your working folder, to subfolder $work/Android-Mergeall, in a file whose name is either:

Whether the logfile name is passed or generated, the logfiles subfolder is created automatically if needed.

You can also launch scripts with & at the end of the bash command (or use Ctrl+z and bg after it starts) to run in the background, so you can monitor output with a tail -f _logfile_.txt. For instance, the following session ensures that Termux is in your working directory by running the gowork command defined in ~/.bash_profile; starts a USB-to-phone update run, saving its output using a generated filename; and monitors script progress:

$ gowork $ bash mergeall-android-scripts/mergeall-plug-to-data-update.sh & $ tail -f Admin-Mergeall/mergeall-somedate-sometime.txt output scrolls here Ctrl+c to exit

Alternatively, the following reports file differences only, saves output to an explicitly named file, and views output only after script completion with the vi text editor (nano also works after an install as noted earlier, you can switch to another Termux session while the merge runs, and you don't really need to gowork again unless you've gone elsewhere):

$ gowork $ bash mergeall-android-scripts/mergeall-plug-to-data-report.sh ma.txt $ vi Admin-Mergeall/ma.txt

The diffall.py Utility

Besides its main merge script, Mergeall includes the diffall.pydeep-comparison utility, and the helper scripts include a script to launch it. The following, for example, compares content copies byte for byte on Android, monitoring output as it appears in a passed-in filename; there is no from and to ordering for drives here, because this isn't a merge:

$ gowork $ bash mergeall-android-scripts/diffall-plug-and-data.sh diffs.txt & $ tail -f Admin-Mergeall/diffs.txt output scrolls here Ctrl+c to exit

As a guideline, you should run this comparison script occasionally to ensure that all files in your content copies are completely identical. Though Mergeall's time-and-size comparisons are fast and adequate for your day-to-day syncs, drives can and do go bad eventually. A byte-for-byte check proves the integrity of your content—albeit much more slowly (read the notebox ahead for details).

Syncs and Diffs in Action

For a more graphic look at the Mergeall sync and diff helper scripts in action on Android, check out the Mergeall-run screenshots with and without tails, and the exemplary but heavily redactedMergeall and diffall.pylogfiles. And for a comprehensive test that verifies the approaches laid out in this document, study the files in this folder, especially its test notes.

Though most phones would lose a straight-up footrace with a PC, on Android, Mergeall itself is relatively fast because it compares just file timestamps and sizes by design. The more exhaustive diffall.pyutility, though, can run much longer for large content collections—which is one reason for using incremental sync tools like Mergeall (though even-slower full copies make it a slam dunk).

As an example, on-phone comparisons of a 150G content collection having 100K files and 10K folders on microSD and USB flashdrive work well, but take 2 hours in diffall.py versus 2 minutes or less in Mergeall. For long-running tasks like the former, be sure to acquire the Termux partialwake lock in Android's notificationsto keep your program running even after your screen blanks. Remember to release it when your task finishes to avoid battery drain.

And for true sports fans: the Mergeall comparison for the same 150G content folder on the same drives clocks in at a quicker 24 and 54 seconds on Mac OS El Capitan and Windows 7 machines, respectively—though the variables are many, and 2-minute on-phone merges still beat microSD-ectomies.

Update: results for internal storage are now in. On the same phone, and using the same USB flashdrive and 150G content collection, Mergeall and diffall.py comparisons to internal storage run in just 19..43 seconds and 29 minutes, respectively. This is _3 to 4 times faster_than microSD's results of 2 minutes and 2 hours on these same tasks. Surprisingly, the cpall.py script copies the full 150G from USB to internal storage in just 21 minutes—slightly faster than a diffall.py comparison, and perhaps indicative of a USB port and/or drive that's a bigger bottleneck than internal-storage write speed. Either way, internal storage takes the gold in the on-phone content sprint.

Update: in fall 2020, a Galaxy Note20 Ultra ran a full byte-for-byte diffall.pycomparison between a USB drive and internal storage in just 15 minutes—twice as fast as the early 2019 Note 9 figure above, despite the fact that the content set had grown to a substantially larger 194G. It's unknown if this reflects faster drives, USB ports, CPUs, or combinations of these; but in computing, _waiting_is often a completely valid work-around.

The FAT Times Fixer Tool

Additional helper scripts in this package run Mergeall's fix-fat-dst-modtimes-data.pyto adjust file modification times on FAT32 drives so they are in sync with non-FAT drives after daylight-savings-time (DST) changes. Because FAT32 is recommended for broadest interoperability on Android today (until Samsung Android 10), you may need to run this utility twice a year, especially if you propagate changes to or from a PC or your phone's internal storage.

Without getting into all the details here, on DST rollovers, FAT32's local timesmay all be one hour off from the UTC-based timesused on non-FAT drives. To diagnose this state, run Mergeall in report-only mode (its reams of messages will show every file differing by modtime), and inspect the times of the same file on both drives (they'll differ by one hour). Less than ideal, perhaps, but cross-platform content management is never dull.

Luckily, this state is also easily remedied. To make FAT32 times match other filesystems, simply run the helper script as follows, with data for on-phone storage and plug for USB drive; an -add or -subfollowed by a number-of-hours value; and an optional explicit logfile name:

$ gowork $ bash mergeall-android-scripts/fix-fat-dst-modtimes-data.sh -add 1 logfile.txt $ vi Admin-Mergeall/logfile.txt

In general, you'll want to run this script after DST rollovers on any FAT32 drives whose times must compare correctly to non-FAT32 drives. Although the possible combinations of filesystems and devices are many, most use cases fall into one of the following categories:

Adjust as needed for your usage, and see Mergeall's user guideand the fixer scriptitself for more details. You can also run the bare fixer script directly, either on your PC (to adjust times on an attached USB drive or a removed SD card), or on your phone (to adjust times on an attached USB drive or a resident SD card); its helper scripts like the one shown in the example above simply make on-phone runs easier. For instance, a direct run may look like this (try Mergeall-source for sourcepath in your phone's $work folder, and use \ on Windows):

$ python sourcepath/fix-fat-dst-modtimes.py folderpath -sub 1 > logfile

On caution here: you should generally restart your phone before syncing content to it after a DST rollover. Otherwise, Android may return file times saved earlier in a cache that are now invalid, and will not compare correctly to other files' times. This is arguably another Android bug; without a restart, its net result may be spurious differences or equivalences, either one of which can derail Mergeall syncs.

For example, after adjusting times on a FAT32 USB drive up by an hour at a recent DST rollover (March 2019 in the US), Android initially and erroneously reported all the drive's times as differing from those on UTC-based internal storage; a phone restart sufficed to make the drive's times agree with internal storage as expected. Restarts may or may not be required for on-phone content on removable drives: at the same DST rollover, FAT32 USB agreed with FAT32 SD and exFAT USB agreed with exFAT SD—without restarts. To be safe, though, restart to prevent Android's error-prone caching from causing problems.

Other Mergeall Tools and Techniques

Mergeall's source-code package also includes the cpall.py folder-copy script, and its test subfolder includes the ziptools package which comes with zipfile create/extract scripts. If you elect to host your content on your phone's internal storage instead of a removable card, for example, you might use either of these to copy your content folder in full from a USB drive initially, or export it to USB later.

Though not wrapped by helper scripts here, both of these tools can be run in your $work folder with normal Unix command lines in Termux, using either your bash-profile settings or custom paths. For instance, both of the following command pairs import and export content copies on your phone ("py" is assumed to be aliased to "python" in your bash profile file here, and routing the logfile is up to you in this mode):

$ py Mergeall-source/cpall.py plugplug plugdata -skipcruft -vv -u > Admin-Mergeall/importlog.txt & $ py Mergeall-source/cpall.py datadata dataplug -skipcruft -vv -u > Admin-Mergeall/exportlog.txt &

$ py Mergeall-source/test/ziptools/zip-extract.py plug/../import.zipplug/../import.zip plug/../import.zipdata $ py Mergeall-source/test/ziptools/zip-create.py plug/../export.zipplug/../export.zip plug/../export.zipdata/* -skipcruft

As usual, tab completion helps with input here, and bash scripts, variables, or aliases can shorten such commands. For more on using cpall.py, see the script; for more on ziptools, see its homepage. Termux itself comes with a Unix unzip and a pkg install zip adds a Unix zip; both generally work well, but are naturally subject to the Unix tools' constraints (e.g., file-size limits may apply).

If you prefer a more interactive experience, Mergeall's console modeworks on Android too. This mode asks you for input parameters, instead of expecting them in the command line. It doesn't have tab completion for paths, though, so it's a bit more typing or pasting, and may work better with a Bluetooth keyboard. See the screenshots of this mode'ssetup and results, and start it in Termux with a command like this:

$ py Mergeall-source/launch-mergeall-Console.py

Finally, you can also skip the helper scripts and bash profile altogether and run Mergeall's scripts directly, as described in its user guide. The following Unix command line, for instance, kicks off a Mergeall USB-to-SD updates run on Android manually:

$ python Mergeall-source/mergeall.py
/storage/7284-2735/Android/data/com.termux/MY-STUFF
/storage/25C9-1405/Android/data/com.termux/MY-STUFF
-auto -backup -quiet -skipcruft
> ../Admin-Mergeall/mergeall-$(date +%Y%m%d-%H%M%S).txt &

And if you can imagine typing this all on a smartphone's touchscreen, you'll probably appreciate why the helper scripts may be a better idea.

Random Tips

This section collects assorted usage pointers and reminders. Most are intended for users newer to Termux, Unix, or both.

Termux Tips

Other Tips

The Mergeall Termux Wrap-Up

In closing, if this Mergeall approach seems like a lot of work, it's because it is, but it's also a price inherent in maintaining content on Android smartphones with Termux command lines today. In exchange, this scheme provides a fully functional incremental-backup and changes-propagation tool that you can run directly on your Android device to synchronize with external drives—no device "rooting" or card juggling required.

That being said, some users of both Mergeall and Android may prefer to remove cards instead of using command lines, and this may be the only option for others. Still, any scheme that can free your content from the constraints of both closed devices and cloud storage is worth a look. Not only are storage devices much quicker than clouds, they can automatically neutralize the proprietary and even exploitive agendas that seem to underlie much of today's smartphone experience.

However you choose to proceed, taking control of your digital property is worth the effort.

Appendix A: A Brief Primer on Android Pathnames

This appendix introduces and summarizes the folders used in this guide. It's probably more useful to readers who haven't used command lines much in the past, but also serves as a glossary of sorts. This appendix was also updated in 2021 for some findings in Android 11 covered in fulloff page; this spoils part of Appendix C's surprise, but makes this primer more useful.

Because the locations at which you'll store your content on Android are central to the content-management approaches presented in this doc, it's important to get a handle on these up front. Android identifies a storage location just like every other platform, with a _pathname_—a string of folder names that reflects folder nesting. Some of Android's naming conventions are unique, though, and some pathnames identify folders with specific roles. Here's a quick rundown on those that play a part in the Mergeall story:

App-private folder

The Termux app used by this guide's approach installs and runs its programs in its Android app-private folder. This folder is fully private to the Termux app: it cannot be accessed by any other apps on non-rooted phones, including file explorers, and is automatically deleted when Termux is uninstalled. This makes it less than ideal for general-purpose storage. The Termux app-private folder is located at this pathname on your phone:

/data/data/com.termux

This folder is really located in internal storage, but is too hidden to be accessed as such. Being Linux-based, Android uses the /character to separate folders in a pathname like this; the one all the way to the left means the device root (the top of the folder-nesting tree), and the left-to-right order of /-separated names gives their nesting. The com.termux part at the end is how Termux is known to Android; other apps have similar dotted-path IDs—and similar app-private folders.

Update: as noted earlier, Termux eventually added support for the Storage Access Framework (SAF), which allows specially equipped file-explorer apps to access the Termux app-private home folder from outside the Termux app. This isn't normal access and subverts Android permissions, but it does allow broader use of this otherwise private folder. For more details, see the later coverage in Android Deltas Sync.

User-home folder

Your Termux "home" folder lives in the Termux app-private folder too, at the following pathname:

/data/data/com.termux/files/home

This is where you normally "are" when you first open Termux (before you run a cd to go to a different folder). It also goes by the names ~ and $HOME, which are just shell syntax that automatically equate to this folder's full pathname above. Because you can't see this folder outside the Termux app, its scope and utility are limited (sans tools covered in the update above). Because of the way Termux works, though, this is where you'll need to install and edit the bash profile file covered in this guide.

Internal-storage folder

Android allows all apps and their users to access and share available space on the phone's internal storage freely (subject to permission requirements for apps that we'll skip here for space). In fact, this space is often called just shared storage in recognition of its cross-app role. Confusingly, Android docs also sometimes call this storage "external" to distinguish it from storage associated with a specific app, but this doc prefers to use that term for physically removable drives.

Thanks to Android history—and the obfuscation magic of Linux links—there are at least two pathnames that can be used to refer to internal shared storage on most devices (but use the shorter unless you happen to enjoy pointless tapping):

/sdcard /storage/emulated/0

Some devices provide alternate paths to this space that are less common. And yes, /sdcard really means internal storage, not your removable SD card, for legacy—and even tortuous—reasons we'll omit here. Because internal storage is accessible to every app, and is never automatically removed (short of a factory reset), it's ideal for content storage if your phone has enough space. It's also generally more convenient than your home folder for working in Termux, because you're not limited to Termux tools.

If you opt to store both your content and working folders in internal shared storage for best utility and permanence, their pathnames might look like the following (though /sdcardmay have to be something else on some phones, and your stuff's MY-STUFFmay naturally vary):

/sdcard/MY-STUFF /sdcard/work

Update: though uncertain at this writing, Android 11 may impose the same update constraints on internal storage that KitKat imposed on removable media (described earlier), making it less flexible than implied here. See Appendix C for more details.

Update: per the most recent news, most of internal shared storage will still be freely accessible to at least some apps in Android 11, but will also run radically slower than internal app-private and app-specific storage, and app-specific folders within internal storage will be locked down to the owning app(more on these folders ahead).

Removable-drive folders

Removable media—including both SD cards and USB drives—show up under the /storage root on Android when they are attached, with a folder name derived from the drive's unique ID and composed of hex digits. For example, here's one for a USB drive with an ID of 25C9-1405:

/storage/25C9-1405

Your drives' IDs will differ (they're supposed to be unique, after all), and you can find them in some Android file explorer apps after drives have been attached (and the system has a chance to mount it in the folder-nesting tree); see earlier for app pointers. A physical drive's Android ID will change if it's reformatted, but should generally remain constant otherwise. Either way, the drive's path allows programs to perform device-agnostic processing of content over USB.

At least that's the removable-drive story on Samsung Android devices, and it's prone to vary across vendors (it depends on how drives are mounted in the underlying Linux system and vendor extensions). USB drives are also generally available at their ID-based folders in /mnt/media_rw too, for instance, but using that path may require special permissions or phone rooting that are beyond this appendix's scope.

Update: per the most recentnews,Android 11 imposes new constraints on USB-drive access, making it less usable than implied here. Though prone to change, general USB paths are revoked, requiring program rewrites to use proprietary Java-based APIs instead of traditional POSIX techniques. See the docfor possible work-arounds.

Removable-drive app-specific folders

Like internal storage, removable drives are accessible to all apps (subject to permission rules we'll skip here too). As covered in this guide, though, Termux does not support updates to a removable drive except in the drive's Termux app-specific folder. This folder isn't app private, but it follows the same pathname format: if you store your content folder on a removable drive, its pathname would look like the following (but use your own drive ID and folder name, instead of 25C9-1405 and MY-STUFF):

/storage/25C9-1405/Android/data/com.termux/MY-STUFF

Every app has one of these app-specific folders if it needs one; they are meant for app-related content that might be too large or inconvenient to store in the /data app-private folder. As discussed in this guide, these app-specific folders are key to using content on removable drives in Mergeall—whether from Termux command lines, or the GUI of the next appendix. Storing content here enables Mergeall updates on removable drives, though this folder's limited scope rules it out for cross-app updates.

App-specific folders like these are also automatically and implicitly deleted when the owning app is uninstalled. This is meant to clean up an app's resources, but makes these folders subpar for storing data. Because of that, you'll generally locate your working folder elsewhere. Your Mergeall content would ideally be elsewhere too, but updatable copies on limited-space phones may have no other option than removable storage, and this requires app-specific folders today.

Internal-storage app-specific folders

For completeness, internal storage hosts app-specific folders too, which follow the same pathname pattern as removable drives, and might look like the following if your content is nested in Termux-specific storage:

/sdcard/Android/data/com.termux/MY-STUFF

This guide downplays this folder, because it's less functional: just like their removable cousins, app-specific folders in internal storage are automatically deleted when the owning app is uninstalled. Internal app-specific folders are also more accessible than app-private space (e.g., file explorers work), but not any more so than the rest of internal storage. Hence, you're normally better off using less-transient locations in internal storage for your content; anywhere else in /sdcardis both safer and just as usable.

When this guide was written, app-specific folders in internal storage were largely lumped together with the rest of internal space, because they did not restrict updates to the owning app—unlike their removable kindred. If this changes (and the following updates suggest it may), it will make app-specific storage folders even less desirable than internal shared storage at large.

Update: though uncertain at this writing, Android 11 may impose new rules on internal app-specific storage, making it less usable than implied here. See Appendix C for more details.

Update: per later news, Android 11 narrows the accessibilityof internal app-specific folders so much that they are generally unusable for content managed by more than one app. Sans special new app permissions, files to be processed in both Termux and Pydroid 3 must be kept in shared storage (i.e., /sdcard/_other_) instead.

Update: Android file-explorer apps ultimately found a loophole which allowed them to provide access to app-specific folders in Androids 11 and 12. Android 13 aims to close this loophole, but the outcome of this is unclear at this writing. See the later coveragein Android Deltas Sync.

Downloaded-content folder

Finally, something simple: documents and packages you download in your web browser normally wind up at the following location in internal storage (or a similarly descriptive locale on other phones):

/sdcard/Download

This is where Mergeall's source-code and android-scripts zipfiles show up if you fetch them from their web pages named in this doc. Because internal storage is completely general and accessible, you can store and use their unzipped content in this folder if you wish (but watch for possible mods to this too in an Android 11 future).

A Storage-Paths Cheat Sheet

As a concise summary of the rules we've just met, Android groups internal and removable storage into accessibility and permanence categories as follows—all subject to the Android device and version variability which defies authoritative docs:

Keep in mind that all of these conventions pertain to Android only. If you use the same removable drive across different platforms, the names of its folders will differ on each. The following, for example, is the pathname for the same folder on Android, Mac OS, Linux, and Windows; all reference the same folder, but you'll need to extrapolate from the Android flavor shown in this guide when using a drive elsewhere:

Android: /storage/25C9-1405/Android/data/com.termux/MY-STUFF Mac OS: /Volumes/EXT256-1/Android/data/com.termux/MY-STUFF Linux: /media/username/EXT256-1/Android/data/com.termux/MY-STUFF Windows: D:\Android\data\com.termux\MY-STUFF

And again, also keep in mind that the pathname forms above and used in this guide may vary on some Android devices, and have varied over Android's history. For instance, /storage/extSdCard and /storage/sdcard0worked in the past too. The paths used here mostly reflect Samsung Galaxy devices using Androids 7 (Nougat) through 10, so you'll have to translate if those on your phone differ. Alas, Mergeall's budget is limited, and stability is not among Android's core virtues.

For more details on Android pathnames, see books, the web, or other Android resources.

Appendix B: The Pydroid 3 tkinter GUI Alternative

User alert: as of its August 2023 version 6.4, thePydroid 3app presented here for running tkinter GUIs is much less usable for programs that share content with other apps. You can still use it to experiment with Python and tkinter without a PC, but its loss of shared-storage access means that you shouldn't expect to use it for most realistic work, and should explore standalone-app builders as an alternative for Python on Android. For the full story, please visit this page.

The main part of this guide recommends and covers running Mergeall by using command lines in the Termux app, because this approach is both straightforward and reliable, and is completely advertising and payment free. Termux provides a full-featured and general-purpose Linux environment, which can run Mergeall's command-line mode with minimal setup tasks and robust performance. This is not, however, the only Python-on-Android game in town.

Most notably, the Pydroid 3app has recently acquired some rudimentary though impressive support for using the Tk GUI library—and Python 3.X's _tkinter_interface to it—in Python programs. This makes it possible to use Mergeall's GUI on Android devices today, along with a host of other existing tkinter software. Less positively, this app's tkinter support also comes with substantial downsides, including both behavioral glitches and unfortunate advertising, that make this approach difficult to recommend in this guide.

That being said, the prospect of running PC GUIs on a smartphone is arguably seductive, and the possibilities that this app raises for Mergeall users merit both fair review and your own judgement. You may or may not prefer Pydroid 3 GUIs to Termux command lines in the end, but this appendix provides enough coverage of the former to help you make a good usage-mode choice.

The findings in this appendix reflect Pydroid 3 versions 2.22 and 3.0;Python versions 3.6 and 3.7; an Android port of Tk version 8.6; and Android versions 7 though 9 (Nougat, Oreo, and Pie). All four of these systems play roles in issues noted along the way. While this how-to hopes to be revised for fixes in these systems, change is constant and rapid in the software world. If you're reading this in the future, be sure to check all three for current status. With any luck, the glitches list here will be completely invalid by the time you see it.

Update: as of fall 2020, Mergeall's GUI has also been verified to run on Pydroid 3 version 4.0, the Python 3.8 used by that app, and Android 10 (formerly called Q). The glitches listed ahead were retested and updated for this context (if not marked "(Fixed)" they're still present). Though not germane to this appendix, Mergeall command lines also run on the latest Termux and its Python 3.8. All told, this means you can run Mergeall and its GUI on Androids 8 (Oreo), 9 (Pie), and 10; Nougat is ruled out by its timestamp bug. Android 11 support is still murky, given its planned change to internal-storage permissions; Mergeall will likely work on 11, but content may have to be nested barring app work-arounds.

Mergeall's GUI in Action

First and foremost, Mergeall's GUI does work in Pydroid 3 with only minor changes, and without the command lines required for Termux use. Under this alternative approach, your content syncs will be run in the Pydroid 3 app's IDE (its edit+run GUI) that looks like this, and the Mergeall GUI you will use to configure and run syncs on your phone will look as follows:

Portrait orientation

Start, browse, report, finish,update,finish

Landscape orientation

Start, browse, report, finish, update,finish

Other viewing modes

Non-maximized,Non-maximized, andNon-fullscreen

You can also view the screenshots above in slideshow mode here. Despite the tradeoffs we'll explore in this appendix, it's important to note that running tkinter programs this way is an enormous achievement, given the large disparities between PC and Android platforms.

Pydroid 3 Viewing Modes

Now that you've seen Mergeall's GUI at work, we need to define some terms that impact the way it renders in Pydroid 3. The first two sets of screenshots above were captured with this app's tkinter fullscreen and maximized viewing modes enabled, by setting the first "Tkinter" switch off and the second on in the app's Settings ⇨ System dialog. You can use neither, either, or both of these modes, but their utility can vary per GUI and usage:

Despite their downsides, Mergeall seems to work best with both these modes enabled, because it has no persistent popup windows and doesn't need to catch program exits, and because screen space is a precious commodity on phones in general. Mergeall works without these modes too—and looks like the last set of screenshots above—but has less space, and may bear manual resizing which seems out of place on a phone.

General Usage Notes

As a more general observation, tkinter GUIs in Pydroid 3 can take a bit of getting used to, and some are substantially easier to operate with a Galaxy Note-type stylus or Bluetooth mouse. For instance:

On the other hand, tkinter GUIs are surprisingly usable on a phone with practice—and might even be compelling.

Programmers may be interested to know that the Pydroid 3 app's tkinter support might be based on the work of AndroWish—an open-source project that uses the SDLgraphics library to render Tk GUIs on Android devices, employs a JNIbridge between Android's Java code and Tk's C code, and was itself based on an earlier project called SDLTk (whose web whereabouts are unknown). If so, this means that Pydroid 3's tkinter simply inherits both its utility and flaws from AndroWish, though the Pydroid 3 commercial product also shares responsibility for the software it bundles and sells.

Unfortunately, Pydroid 3's developer did not reply to requests for verification of AndroWish use. To be sure, Pydroid 3's tkinter bears multiple signs of AndroWish-ness. Among them: its two-finger zooms and three-finger zoomed pans work in AndroWish too; its unique rightclick bindings smack of AndroWish's sdltk touchtranslate command; its GUIs-cannot-spawn-GUIs constraint discussed ahead also exists in AndroWish; its tendency to fire multiple events for one gesture parallels an AndroWish bug report; its install tree includes SDL components used by AndroWish; and its overall look-and-feel seems the same as AndroWish's. Still, this isn't enough evidence for a positive identification.

What can be said is that Pydroid 3's tkinter is likely based on work of others that is not necessarily tied to Pydroid 3, and might be usable in other Python coding apps, in standalone apps of your own, and even on other platforms. While intriguing, such prospects are both beyond this guide's scope, and largely moot. The more important point here is that Pydroid 3 today brings Tk and tkinter on Android to the broad Python audience—something that the Tcl-biased AndroWish project seems unlikely to have ever pursued. Hence, we'll attribute Pydroid 3's tkinter to Pydroid 3 here because it's no less a part of this app than any other component it embeds, and note that running tkinter GUIs like Mergeall's on Android is a stellar achievement, regardless of its implementation.

Update: after a brief tour of build options and code, it appears that the SDLTk library may be all that is required to enable tkinter in Python scripts on Android (along with the SDL library which SDLTk uses, and substantial Java-related anguish).AndroWish by itself adds extra utility for Tcl programs, which may not be as valuable to Python code. That said, the most usable SDLTk version available todaywas also updated and posted by the developer of AndroWish, so it seems reasonable to call the two a set. Given SDLTk's separate origin, though, it also seems reasonable to call Pydroid 3 a sort of AndroWish for Python.

Mergeall Pydroid 3 Requirements

The prior section paints a justifiably positive picture, but tkinter GUIs do not come without both procedural requirements and noticeable drawbacks in Pydroid 3. To bring Mergeall's GUI to life on your phone, you must first be willing to accept some constraints unique to Pydroid 3, and others common to Android in general. Specifically, you must:

Compared to the Termux approach, you do not need to install Python separately (it's part of Pydroid 3); install helper scripts (you'll use the GUI instead); or run a permissions setup command (Pydroid 3 requests permissions automatically).

However, you still need Mergeall's source-code package (you'll run its GUI's script); must patch two of its code files for Pydroid 3 alone (per the next section); and still have to work with Android version, filesystem, and permission constraints (per their earlier coverage).

While some of the steps above are self-explanatory, and we don't have space for an exhaustive dissection of the rest, the remainder of this appendix explores the most-salient points on the list above.

Required Mergeall Code Changes

Among the requirements just listed, the third may merit explanation most. To use Mergeall's GUI in Pydroid 3, you must first download Mergeall's source-code package from this page to any writeable folder on your phone, and unzip (i.e., extract) it there. Internal storage's /sdcard/Download is the usual target for downloads, and will suffice for both a quick look and later syncs. There's more on Mergeall's source-code package here and hereif you've jumped into this appendix randomly.

After installing the Mergeall source-code package, you'll need to replace two of its files, because it's not possible to integrate Pydroid 3's changes into the source-code package today for reasons explained ahead. To do this, simply click the following links to fetch custom versions of two Mergeall source-code files, and move or copy them into the top level of your unzipped Mergeall source-code package folder on your phone. That is, replace the unzipped originals with the versions below:

If you click these links online, you'll receive reply pages that looks like this; download to your phone by clicking the "save" in these pages' "Raw text" lines. If you're working offline, try a rightclick/longpress save, or visit the folder these links reference.Tip: some browsers may append a bogus ".txt" to the end of a saved ".pyw" file's name; delete the ".txt" manually by renaming if needed.

What Was and Wasn't Changed

The changes applied in these files are minor, and apply to Mergeall's GUI only. Hence, they won't break Mergeall's command-lines mode described earlier in this guide if it's run from the same modified source-code folder. Conversely, these changes will break Mergeall on PCs, so use the files above only on your phone's copy of the source-code package. All changes made are marked with "# ANDROID" comments; search for this string in the custom files if you want to see what differs on Android.

In short, these files' changes address some but not all limitations of tkinter GUIs in Pydroid 3. For example, text labels were trimmed and control fonts shrunk for better fit; some size settings were lowered for non-maximized mode; a workaround was installed for a Pydroid 3 sys module glitch that prevented the GUI from spawning merges; other workarounds were installed for Python webbrowser issues that precluded document views; a button was decolorized to avert color loss; and later releases added dialog-text breaks, folder-choice prefills, and read-only text state, to avoid message truncation, chooser dialogs, and keyboard popups, respectively.

Despite all the changes applied for Android, though, you must still avoid using some fonts in customizations because they trigger crashes prior to Pydroid 3 3.0 and force blind choices later, and should not rotate some devices while Mergeall is running because this can hang or kill Pydroid 3 altogether (until its version 4.0).

We'll take a closer look at these and other glitches in Pydroid 3's tkinter GUIs in a moment. First, though, let's run the GUI.

How to Run Syncs in Mergeall's GUI

Once you've made the prior section's code changes, start Mergeall's GUI by opening its source-code package's file launch-mergeall-GUI.pyw in the Pydroid 3 app's editor, and pressing the editor's big yellow run button.

Mergeall's GUI shouldappear, and work the same as it does on PCs and as described here—select FROM and TO folders, choose a REPORT or UPDATE, pick a logfile folder and other run options, and press the GUI's GO button to sync. Mergeall messages scroll in the GUI, and are saved to a logfile if you've requested one. As usual, you can open Mergeall's source-code file in either Pydroid 3 itself (see its folder icon), or most Android file explorers (tap the file and open with Pydroid 3).

Configuring Pydroid 3

As noted earlier, Mergeall tends to work best in fullscreen and maximized modes. If desired, select these modes before running the GUI with the Settings ⇨ System dialog, and press your phone's back button twice when you wish to exit Mergeall in maximized mode. While you're in Pydroid 3's Settings, also consider using the Editor's monospace font without line wrapping; Mergeall's code is otherwise difficult to view. Viewing code is not required for the GUI, but you may wish to experiment with configuration-file settings in mergeall_configs.py, and the line between user and programmer is often gray at best.

Update: two related tips here. First, unlike Termux's run-on-demand tool, Pydroid 3's extra-permissions request popup has historically been triggered by a first permission error; if Python scripts generate permission errors in Pydroid 3, it may help to create a new file in the app's IDE and save it to a folder outside Pydroid 3's own, to force the permissions request. Also, there's a new sidebar in the companiontkinter docwith info on setting Pydroid 3's DPI settings; this can improve GUI-to-display fit.

Other Mergeall Tools

This GUI approach works, but only for Mergeall syncs: other Mergeall tools like diffall.py comparisons and cpall.py bulk copies have no GUI, and are still best run with command lines in the Termux shell as coveredearlier. Pydroid 3 does have a Terminal option that is capable of running command lines too (and can even run the main Mergeall script with a command typed manually), but it's much more limited than Termux for command-line work. And while you can open and run such scripts in Pydroid 3's IDE, they won't work because the IDE has no way to specify the command-line arguments they require.

As a special case, running Mergeall's GUI to sync to an empty folder has the same effect as cpall.py (and in fact uses the same code), but you won't get as many status messages, and there's no GUI equivalent to diffall.pytoday. Command-line scripts need command lines (or an IDE that supports them).

Opening and running a source-code file in Pydroid 3's IDE normally requires multiple steps—you must either open the app and navigate to your file, or navigate to your file in a file explorer and open with the app. This, plus clicking the IDE's run button, makes for substantially more steps than either a command in Termux or a standalone app (and is one reason this guide downplays the GUI option).

That said, some Android file-explorer apps allow you to place a shortcut (a.k.a. link) to a source-code file on your home screen, like this andthis. Once placed, starting a tkinter GUI program like Mergeall in Pydroid 3 is just _two taps_—click the shortcut, and click the IDE's run button. Pydroid 3 still has extra setup steps and glitches, but running a tkinter GUI can be nearly as easy as running an app. Usage notes: file explorers like this are among those that make links, and be sure to set the default app or command for Python files (both .pyand .pyw) to be Pydroid 3 in your file explorer before creating the shortcut.

Update: if you use the latest 3.0 version of the Total Commander file explorer noted throughout this page, be careful to uncheck the "Parameters" box when making internal associations and home-screen shortcuts. Else, the content:// URIs sent to Pydroid 3 fail to open the target Python source file. It's unclear if this is a Total Commander or Pydroid 3 issue, but the file:// URLs sent by the former's pre-3.0 versions still work well on Android 10 in the latter's current 4.0 release. This doc formerly also noted ES File Explorer Pro, but that app was removed from the Play store for advertisingviolations; good thing Google sacks this sort of thing...

Glitches in Pydroid 3 tkinter GUIs

Now that you know how to run a GUI sync on Android, you should also be aware up front that some of Pydroid 3's tkinter support may still not quite be ready for prime time. Running tkinter GUIs on Android is undoubtedly a major accomplishment, but the current implementation still has rough edges that Mergeall code changes cannot entirely smooth. In the interest of full disclosure, this section covers the most prominent technical glitches known to confront Pydroid 3 tkinter GUIs.

Three brief scope notes about the items described here: some don't impact Mergeall, but do impact other programs; some are probably of more interest to programmers than to end users, though the distinction for source-code programs is gray; and some have been repaired as of this doc's latest update, and are marked "(Fixed)" in this list. Also note: if JavaScript is enabled in your browser, a "Show Index" button below toggles a succinct index list summarizing this section's content:

GUIs may require redesign for fit

Some PC GUIs may be too large to fit well in the smartphone form factor, and require changes to avoid widget truncation in maximized mode, or horizontal slides in non-maximized mode. In Mergeall, this meant trimming text labels and shrinking control fonts for better fit on smaller screens, and lowering some preset size configurations for non-maximized mode (though they're irrelevant at maximized size).

Getting the fit right for smaller phones can be difficult. For example, because Pydroid 3's tkinter naively clips nearby widgets instead of truncating the text of labels that don't fit, Mergeall's original and descriptive PC-GUI labelswere initially unusable on phones of all sizes, and had to be trimmed radicallyfor use on screens as small as 5.5" (and may still be too wide on even smaller displays without new user configs).

More-sophisticated tkinter GUIs may naturally require more redesign work. This is especially true for those that open multiple active windows at once—a desktop-metaphor paradigm at odds with both Pydroid 3's rendering options and Android's one-window-at-a-time app model. While new tkinter GUIs can be designed with such models in mind, many existing GUIs' shrinkability will be firmly limited by usability; Mergeall might someday run on your watch, but it probably shouldn't.

Update: it's worth adding that Android GUIs can also be impacted by global user settings for font size, screen zoom, and display resolution; GUIs have no control over these, and might not accommodate pathological cases. Pydroid 3 tools such as maximized mode and low DPI settings can often improve rendering, but users share some responsibility on this front. For tips on better fitting GUIs to displays, see the related doc's sidebar.

(Partly fixed) Font support is limited and crashy

Font support in Pydroid 3's tkinter is weak enough to ensnare many PC GUIs, including Mergeall's. In general, font-family names courier, times, and helveticawork (and monaco is courier, and arial is helvetica), but most other font family names trigger an immediate and silent hardcrash (including system). Moreover, font _sizes_work as expected, but italic and bold font styles are simply ignored if used for courier (though they work for other font families). To test this for yourself, edit and run this filein Pydroid 3.

Mergeall avoids crashes with font presets tailored for Pydroid 3 to use only supported fonts, but this constrains functionality, and is error prone in the context of end-user settings. You can still tailor its fonts in the configurations file, but customize with care (and see this filefor names to use with platform-specific settings like fonts).

Update: Pydroid 3's 3.0 release in early April 2019fixed its font crash by coercing any unknown font-family name to helvetica, but its other font limitations described above are still present. That is, unknown fonts no longer crash GUIs silently, but helvetica is substituted blindly, and no additional family names (or courierstyles) are now supported. It's a fix of sorts, but the app's tkinter clearly requires more font work.

(Fixed) Phone rotations may kill or hang GUIs

Even more glaringly, on some devices, rotating the screen between portrait and landscape orientation while any tkinter program is running either kills the GUI immediately and silently, or fully hangs the Pydroid 3 app so badly that it must be forcibly killed and restarted. Whether you get a kill or hang depends on which GUI you run, but both effects are clearly less than ideal.

Due to testing constraints, the exact scope of this issue is unknown. It has been observed on Samsung J7, S8+, and Note9 devices running Android Nougat, Oreo, and Pie. On the more recent Samsung Note9, however, rotations in Pydroid 3 tkinter GUIs work fine in both Oreo and Pie after changing the display's screen-zoom setting from Small to either Medium or Large. This doesn't help on the older phones, though, and it's unclear if the older phones may benefit from applying system updates.

It may also be possible to work around this in Mergeall with a Java-based approachthat locks orientation or ignores rotation, but this would reduce utility drastically (rotation can be an essential tool when space is limited), and the underlying bug should really be fixed globally for all programs. Ultimately, only Pydroid 3 tkinter GUIs exhibit this defect.

For now: if rotates don't work in tkinter GUIs on your device, either make sure your screen zoom is not Small and your device is up to date; wait for a fix in whatever software plays Brutus to Pydroid 3's rotation Caesar; or use Mergeall's GUI in either portrait or landscape orientation—_only_—and consider a rotation-locker app like one of theseif you can't avoid flipping your phone.

Update: the screen-rotation issue described by this section has been fixed as of Pydroid 3's 4.X release in 2020. Specifically, screen rotations have been verified to work well and without requiring screen-zoom settings when using Pydroid 3 version 4.01 on Samsung Android versions 7 (Nougat) through 10. Please upgrade your Pydroid 3 app as needed to avoid this issue.

(Fixed) Phone rotations during activities misplace or hang GUIs

In the same department as the prior note, changing phone orientation during a spawned activity may cause a running Pydroid 3 tkinter GUI to slide off screen, or hang similarly. Slides happen even on phones that support GUI rotation in general, and have been observed after rotating both Sharesand document handlers opened by the GUI (e.g., web-browser help displays). Rotations are clearly a Pydroid 3 tkinter problem area; as a workaround, rotating twice generally restores a slid GUI—but only on devices that support GUI rotation.

Update: the screen-rotation issue described by this section has been fixed as of Pydroid 3's 4.X release in 2020; see the preceding item's update for more details, and update your Pydroid 3 to avoid this issue.

Python's webbrowser works marginally in Pydroid 3's 2.2 release

Python's webbrowser module for opening web pages and other content generally works in Pydroid 3, because the app presets the BROWSER environment variable to an Android amactivity-manager command-line template; when spawned, this template opens documents per the Android default-apps model. See the module's documentationfor more on why this works, andthis scriptfor a demo of the technique. It's similar to open and startcommands on Mac OS and Windows, respectively; on Android, chosen viewers run in parallel with their opener, and a back tap returns to the spawner immediately. Documents can also be opened with Python's os.system and the BROWSER setting, but that apes what webbrowser does automatically.

Even so, webbrowser is weak enough on Android to classify as a glitch (and initially warranted disabling widgets that rely on it). For one thing, it requires local-file URLs to use a file:// prefix on Android, unlike any other platform—and silently fails to open if this is absent. For another, Android's default-apps model is not as functional as file associations on other platforms, and reliance on this limits webbrowser's utility: as tested, text files open with few and unconfigurable handler options, and local HTML files open as raw HTML in a text editor instead of being rendered by a web browser. Barring an unknown Android trick, opening HTML in a web browser requires an http:// server URL, which limits HTML documentation to online use only.

In Mergeall's GUI, these limitations impact the "Help" and "Popup logfile?" buttons, which display the user-guide HTML and run-results text, respectively. These widgets were disabled in earlier releases requiring users to open documents outside the GUI, but now work as well as they can. To work around the local-HTML issue, Mergeall now opens its user guideby remote URL; this requires online access, but has the advantage of always displaying the most recent version (HTML documents tend to change as frequently as web browsers). To display logfilesproperly, Mergeall now uses file:// prefixes to open per the default apps on your devices; this may sometimes be too limited to be useful, but it's better than disabling the feature altogether.

These workarounds were also adopted on Android by other programs mentioned in this guide, all of which now open their help and other documents from within their GUIs, just as they do on PCs. For coding details, see Mergeall's patched files, as well as those of Frigcal and PyEdit describedhere andhere. Caveat: given webbrowser's limitations, Android users might still be better served by manually opening documents in some cases. Caution: per the prior note, Pydroid 3 tkinter GUIs may slide off screen or hang if phone orientation is changed during a spawned activity; rotate twice to restore the GUI where possible.

Update: this story was rewritten in light of new findings in mid-April 2019, but was modified yet again by new bugs discovered later in Pydroid 3's 3.0 release—a plot element large enough to warrant its own coverage in the next note.

Python's webbrowser is fully broken in Pydroid 3's 3.0 release

_Stop the presses_—as of early April 2019, Python's webbrowsermodule is now fully broken on Android due to changes in release 3.0 of Pydroid 3. Document opens using this module work as described above in the prior 2.2 version of Pydroid 3, but they either fail with permission errors or are deliberately disabled in the latest versions of this app, and require additional workarounds applied to programs here on April 21. This complex breakage reflects two separate regressions in Pydroid 3:

Luckily, there are at least two coding workarounds for the damage to other programs: manually unset DISPLAY to reenable webbrowser in general and reset BROWSER to its pre-3.0 activity-manager command for file URLs; or avoid this module altogether by spawning the activity-manager command with os.system (again, see this demo). All former webbrowser clients here were updated to use the latter scheme because it leaves environment settings intact for other Pydroid 3 use cases.

Perspective: although this workaround suffices for now, this episode puts the app's viability as a host for application programs like Mergeall on shakier ground. The tale of this bug has been rewritten multiple times here already, and new webbrowser failures introduced by an "upgrade" of Pydroid 3 that preferentially aids one program at the expense of all others seem to tip the scales towards unusability—especially given the lack of notification to paying customers. At some point, prejudiced instability can be neither documented nor accommodated. That point may remain uncrossed, but seems clearly in sight in 3.0.

Update: as of Pydroid 3's 4.0 release, the DISPLAY environment variable is no longer set, but the 3.0 Jupyter-biased kludge remains in place, leaving Python's webbrowser module still broken for all other use cases through Android 10. Fix, please!

Python's multiprocessing doesn't work on Android (but threads do)

While not a Pydroid 3 flaw and irrelevant in Mergeall, Python's multiprocessing parallel-execution module does not work on Android, because required semaphore tools are not supported on this platform (dig deeper here and here). The impact of this will naturally vary per program. PyEdit's Grep command, for instance, depends on this module on PCs to run searches in parallel with the GUI; while it can be easily configured to use fully supported threads on Android instead, doing so makes it impossible for pure Python code to leverage multiple CPU cores—a speed factor whose significance on phones is unclear. Moreover, PyEdit benefited from preexisting code that leverages the API similarity of threads and multiprocessing; other programs may require more extensive recoding.

(Fixed) Pydroid 3 breaks spawns by leaving sys.executable empty

Mergeall's GUI runs the main work of merges as a separate process by design (this lets you run merges outside the GUI too). Oddly, Python's sys.executable is left empty by Pydroid 3, which in turn makes the subprocess.Popen process-spawn code in Mergeall's GUI fail. Mergeall works around this by hardcoding this setting to use the path to Pydroid 3's Python, but this should be fixed in Pydroid 3: this same bug crippled most of the larger programs tested, though some other programs' spawns were also hobbled by the next issue. The workaround's change also applies to the Pydroid 3 app only, which makes it impossible to integrate into base files (more on this ahead).

Update: Pydroid 3's 3.0 release in early April 2019 did not fix its sys.executable bug (this setting is still empty), but it did move its Python executable to a different path—and break Mergeall's initial workaround to this bug in the process. See the new exception here. Mergeall's updated workaround addresses this in its April 19 patch-code release, by using the result of a spawned which python shell command, instead of former or current paths to Pydroid 3 Pythons (see the demo code). The result is release and path agnostic, and will hopefully be immune to future morph in an app which today seems more likely to sprout packaging-structure changes than fix reported bugs.

Update: the Pydroid 3 bug described in this notehas been fixed. Pydroid 3 doesn't post a formal changes log, but sometime around its 4.0 release in 2020, this app was fixed to properly set sys.executable to the running Python interpreter's path in tkinter GUIs. This attribute was left empty in earlier releases, necessitating the initial hardcoding and later which work-around of the preceding update. The which work-around is harmless, but is no longer required—except on devices running older Pydroid 3 versions without the fix. For an example of how to apply the work-around only when needed, see the updateddemo code.

Two postmortem notes about this glitch. First of all, it was later discovered that Pydroid 3 left sys.executable empty in 2.2 and 3.0 _only_for programs that included a statement which imported tkinter, and were thus identified as GUIs. This was true even if the import statement was preceded with a # to stub it out as a comment. Programs without an import, or a comment resembling one, had a correct Python path. A naive scan for imports is enough to invoke much magic in Pydroid 3—both dark and otherwise.

Secondly, though it's impossible to be certain without access to Pydroid 3's build system, the prior update's Python move in Pydroid 3 3.0_may_ reflect either Android NDK changes, or the Android app installer's selection of an ABI-specific variant from a fat APK (i.e., different versions of the compiled code may imply different paths on different devices). Regardless of relocation causes, though, the Python interpreter's path should have been—and now is—reliably stored in sys.executable; else, a variable path cannot be accommodated without work-around heroics like those deployed here. More Android ABI detailshere and here.

GUIs cannot spawn GUIs

Though it doesn't impact Mergeall, tkinter GUI programs can spawn other tkinter GUI programs in Pydroid 3, but the spawned programs' windows never appear for reasons unknown. Both Frigcal's and PyGadgets' launchers, for instance, start programs that run (e.g., they can write to files) but remain invisible, even though Mergeall's GUI successfully spawns a non-GUI process the same subprocess way after the prior note's patch. This is fatal to programs like PyMailGUIthat cannot be used without their launchersand cannot be easily recoded to use top-level windows instead of freestanding processes (more details here).

While its exact cause remains a mystery, this restriction may be inherent in Pydroid 3 if its tkinter structure requires programs to be run by its IDE. Tellingly, tkinter programs also fail if run from Pydroid 3's own Terminal interface, and the IDE seems to scan specially for specific import statement forms that enable GUI support (see the next note). Moreover, AndroWish—the most likely underlying Tk implementation—appears to have similarconstraints. If a workaround exists at all, it is hidden so well that it's effectively nonexistent.

Pydroid 3 is picky about imports

Pydroid 3 must see a script's tkinter import or fromstatement on a line all by itself, or the script's GUI won't work (the required scaffolding appears to be unloaded). This didn't require a change in Mergeall, but the PyToe program in PyGadgets did not open in Pydroid 3 until such a specific statement was added to its top-level script. Though less common, you may also need to add a tkinter import statement to programs that load and run tkinter code from files dynamically; an import in the loaded code won't be enough to enable tkinter support.

Pydroid 3 crashes badly on larger source files

And now for something completely unnecessary: Pydroid 3's editor has a hard—and arguably naive—limit on the sizes of files it can open, and won't even warn you when the limit is breached. Instead, if you open a flawless script whose source file is larger than roughly 256K bytes (or 5K lines), its runs will fail with Android error-message popups stating that Pydroid 3 has stopped; exiting the script's GUI may exit Pydroid 3 altogether; and the Pydroid 3 IDE itself may start failing randomly but regularly with the same messages. It's unclear if bytes or lines kills the IDE, but large files seem its Achilles' heel.

This surfaced when adding code to the main script of the PyEdit program: a 256,898-byte and 5,141-line prior version worked fine, but a 258,031-byte and 5,160-line new version started failing for no reason whatsoever—until wholly unrelated comment lines were removed (a short launcher script's success was the first clue of size issues). Given today's memory capacities, this size limit is absurd; people do write 5k-line program files, and tools do generate them. It also has no workaround; if your script starts failing inexplicably in Pydroid 3 after adding new features, make sure its code file is under 256K bytes (or 5K lines) in size—or write a two-line launcher that reads and execs the file to subvert Pydroid 3's draconian constraint.

Double-back exits don't run exit handlers

Pydroid 3 has a double-back-click option which exits a GUI, and is often required in maximized viewing mode due to the absence of a window-border exit button (the top-right corner's "X" in windows like this). Strangely, double-back exits close GUIs immediately without running their exit handlers. While such silent exits have little consequence to users in Mergeall, they can be catastrophic in other programs that must finalize work on closes, and are at best a use-at-your-own-peril feature.

The PyEdittext editor, for example, has an explicit Quit button that works normally, but has no way to offer users a chance to save file changeson double-back exits. Worse, the Frigcal calendar has no explicit quit widget of its own, and hence no way to save calendars in maximized mode. Non-maximized mode does better with a window-border exit button that runs exit handlers properly, but doesn't use available space as well, and can still be the scene of a silent-but-deadly double-back exit. The lesson here: don't use a double-back in Pydroid 3 unless doing nothing is what you mean to do.

Update: Pydroid 3's 4.0 release in 2020 seems to have changed double-back exits to be single-back exits: just one system-back tap now suffices to close a GUI on Samsung devices tested. This seems inadvertent (it's still dubbed "double" in option menus), and may be a new buglet; either way, exit handlers are still skipped on the exit—perilously.

Newlines may need help

Though not a factor in Mergeall, pressing the return (a.k.a. enter) key on some Samsung on-screen keyboards does not move the cursor to a new line in text widgets of tkinter GUIs in Pydroid 3. For reasons unknown, the return key's newline character never makes the trip between keyboard and widget, which renders general text-input components virtually unusable. This bug may be limited to specific versions of the Samsung keyboard—it's been observed only when using version 3.2 on Android Oreo, and both older and newer versions' return keys work correctly. But where returns fail, they appear to fail in Pydroid 3 tkinter GUIs exclusively.

Luckily, there are at least three ways to work around this glitch if it impacts your device: use a physical Bluetooth keyboard; use Samsung's handwriting input option whose return key works correctly; or simply install an alternative on-screen keyboard whose return key works properly, such as Google's Gboardor Hacker's Keyboard. Both alternative keyboards support multiline-text input in programs like PyEdit and Frigcal, and the latter keyboard also helps with the next glitch on this list.

Keypresses may need help

Though this also doesn't impact Mergeall, and isn't a bug in any of the systems involved, PC GUIs designed to expect a keyboard may require redesign or logistical workarounds when used in Pydroid 3, because the on-screen keyboard may not be readily available. The PyPhoto program, for example, requires keypresses to process photos, and its main display has no text input field to tap—the only way to force most preinstalled on-screen keyboards to appear.

To work around this, use a Bluetooth keyboard, or install an alternative on-screen keyboard on your phone that can be opened on demand.Hacker's Keyboard, for instance, includes a permanent-notification option which allows you to pop up its on-screen keyboard at any time by clicking the app's entry in Android's notificationspulldown. The resulting keyboardis a complete solution in PyPhoto, and helps with newlines, menu shortcuts, and copy/paste in other programs too (see the notes above and below).

Keypresses may need silencing

Of course, too much of a good thing can also be a bad thing: Android's automatic on-screen keyboard popup may be an unwelcome feature in some Pydroid 3 tkinter GUIs. Interacting with any text field mindlessly triggers the popup—unless it's silenced by code changes which mark text fields as uneditable, with Tk's readonly or disabled states.

The Frigcal program, for example, required readonly state in Tk entry widgets to mute annoying keyboard popups on most touch gestures that made the program unusable. Mergeall's GUI wasn't impacted quite as badly, but its message-text area still required read-only state on Android (which means disabled in Tk's text); otherwise, slower swipes to scroll trigger an on-screen keyboard popup—which overlays and obscures the very text you're trying to view. Despite the work, keyboard popups may still appear more often than they should in some Pydroid 3 tkinter GUIs; as usual on Android, your device's back button (and a downswipe where supported) hides an on-screen keyboard that's obscuring important content or GUI components.

Keypresses botch shifts in some programs

A technical keyboard issue not relevant to Mergeall but a potential pitfall to others: Pydroid 3's tkinter also fails to send expected details to general keypress handlers for all physical and on-screen keyboards tested. For instance, the return and backspace keys send tkinter event objects with an empty char attribute and a keysym that identifies the press (as 'Return'and 'BackSpace', respectively). This differs from other platforms but can be worked around with binds to these specific keys (e.g., to <Return>).

Worse, though: shifted keys, like + and *on most keyboards, send two useless keypress events—one for the shift key itself, and another for the unshifted key pressed. A Shift+* sequence, for example, sends a shift followed by the unshifted 8 key only; the * never comes through. Binding to specific keys doesn't help in these cases, and mapping unshifted to shifted keys manually seems implausible across all keyboards and usage contexts. Barring a workaround as yet unknown, this makes shifted keys unusable in some tkinter GUIs run on Pydroid 3.

This glitch seems unrelated to the newlines issue noted earlier (which fails on just one keyboard), and on the upside appears limited to general keyboard handlers (tkinter text-input widgets process shifts correctly). But it impacts programs likePyCalcthat allow numeric operator keys to be entered on keyboards. As is, PyCalc fully supports keyboards in its "cmd" popup input field (whose use also seems to oddly correct the shifted-keys bug), but keyboard input for its main display is limited to non-shifted keys on Android alone.

Copy/paste may need help

Android-style copy and paste via longpresses is not available in tkinter GUIs run in Pydroid 3. Instead, to copy and paste text between any text-input widgets: first, highlight text with touch, stylus, or mouse; then either use program-specific widgets (e.g., menu options) where available to copy and paste, or use the PC-like key combinationsCtrl+c to copy and Ctrl+v to paste in keyboards that support these. You'll find support for these keys in both Bluetooth keyboards, as well as alternative on-screen keyboards likeHacker's Keyboard(yes, a pattern is emerging). This works well, but may not be what a typical Android user would expect.

Buttons glitch #1: losing background color

The fundamental Button widget in Pydroid 3's tkinter has a pair of issues that seem to reflect state problems and do not respond to tkinter update calls. Only one impacts Mergeall and the other is narrow in scope, but both merit a heads-up for both users and programmers.

Most noticeably, on every Android device tested, Buttons lose their bg background-color setting when pressed: their color changes to the recessed default, and stays that way until anything else on the GUI is tapped. This can seem to make a Button appear pressed, but is really just background colors being lost. Curiously, some buttons evade color loss (e.g., those that open common dialogs may be immune, suggesting a timing issue), but most colored buttons are impacted.

This colors glitch affects only the "Help" button in Mergeall, but may afflict other GUIs more broadly. The only known workaround requires changing code to either accept the Button's default color, or replace Buttons with Label widgets that use click bindings. Labels retain their background colors when clicked, and their cosmetic and functional difference is trivial (Labels don't give press feedback, but Buttons in Pydroid 3 sometimes don't either). As fix examples, Mergeall eventually chose default colors; the PyCalccalculator program adopted Labels to skirt the issue; and the Frigcal and PyToeprograms have used Labels all along (since their ports to Mac OS, which has similar button-color limitations).

Buttons glitch #2: getting stuck

More rarely, Buttons may actually become stuck in a pressed state, in which they remain visibly recessed, and may respond to a future press by triggering two press events at once. Luckily, this glitch appears to be very limited in scope; it has been observed on only one device running Android Nougat (a Samsung S8+) and only occasionally on this device, and has not been seen on any other devices tested—including Samsung devices running Oreo and Pie, and an older Samsung J7 also running Nougat. The flaw trigger on the S8+ is a complete unknown (like much in Android forensics), and there is no known workaround. If your tkinter buttons are truly stuck "on," tap elsewhere in the GUI to "unpress" them.

Android issues cast a wide net

Although not specific to Pydroid 3, the Android issues we've seen besetting Mergeall afflict other tkinter GUIs too. In theFrigcal calendar GUI, for instance, calendar updates are limited to internal storage or app-specific folders on removable drives, and work only on Android Oreo and later due to the same timestamp bug that constrains Mergeall. Violating these rules produces the respective crash scenes here andhere. In Frigcal, the timestamp bug kills calendar backups and makes it view-only before Oreo. In other GUIs, Android's bugs and permission rules may limit functionality similarly.

And so on

This appendix cannot provide a comprehensive list of Pydroid 3 tkinter defects—and it's impossible to report them to lists elsewhere, or even reference such lists, when the underlying implementation is unknown—but testing performed so far has uncovered additional pitfalls which merit brief cautions here. In the other-issues bucket, Pydroid 3's tkinter also:

Probably the best spin on these is symptoms of still-maturing software.

There's more on the other programs noted in the preceding list in the next section. The point here is that while some accommodation is to be expected when translating between PC and smartphone paradigms, issues like those above mean that running tkinter programs in Pydroid 3 is hardly seamless. Even allowing for fixes in its future releases, you'll clearly want to test-drive this app on tkinter programs you'd like to use to see how its support applies.

Update: Later testing on later Androids uncovered new and additional Pydroid 3 tkinter glitches—including doubled characters for shifted keys (on both Android 10 and 11 Samsung devices, and perhaps all Pydroid 3 4.0); and permission errors when spawning activity-manager commands that start intents (just in Pydroid 3 on Android 11). Android 11 also revokes USB-drive access and slows to a crawl programs that process files in shared storage (like Mergeall), but these are not Pydroid 3's responsibility. For the whole story on these later glitches, see theAndroid 11 updates page.

The Advertising Scourge

Despite the workarounds and glitches of the preceding section, the second to last of the Pydroid 3 requirements listed earlier is really its most regrettable and grievous downside: the free version of Pydroid 3 will flash distracting and distasteful fullscreen ads in your face every few runs, unless and until you pay a required $9.99 freemium fee.To be blunt, this comes off as ransom, and borders on extortion. It may be a norm in Android's culture, but it is rude and unacceptable in Python's, and makes it difficult to recommend Pydroid 3 to Mergeall users today.

Together with the prior section's glitches, this also makes it impossible to integrate the changes required for Pydroid 3 tkinter GUIs into source-code package base files. For example, the coding workarounds that accommodate Pydroid 3's font limitations and webbrowser breakage are specific to Pydroid 3 alone, and the sys.executable fix required for spawns may not work in other apps. Changing portable source-code packages to support just a single app on Android is ethically dubious at best—and just plain wrong when that app employs advertising. Though less convenient, Pydroid 3 today is best supported with the custom source-code files used here.

This guide's broader advice to Mergeall's users is to evaluate its GUI in Pydroid 3 to see if it warrants the fee in your context. It may, especially if you have other tkinter programs you wish to run on Android. As examples of this app's broader scope:

See this pagefor a separate how-to on running these programs on Android too, along with a summary of this appendix's Mergeall GUI coverage. Taken as a set, programs like these and others may justify the ransom's price, but this choice is yours to make.

Wherever you may land on this issue, there's no denying that Pydroid 3's advertising scheme casts programs like Mergeall in a supporting sales role that's both uncomfortable and unseemly. A free trial period would be much more palatable than obtrusive advertising, and will hopefully be a part of the future of the otherwise-remarkable Pydroid 3 app. Barring that, we'll have to eagerly await the Android debut of free and open-source alternatives for running the free and open-source tkinter.

Naturally, there may be other tkinter-on-Android options afoot or unknown at this guide's publication, and others may appear in the future. Please exercise the usual due diligence, and check back here for updates as change warrants and time allows. This guide wishes to be inclusive, but some commercial projects may be no more transparent than they are benevolent, and the quality of web-search results has grown inversely proportional to the avarice of companies providing them.

General Requirements Still Apply

Finally, in addition to all the Pydroid 3 requirements described in the preceding sections, you must also keep in mind that this appendix's GUI approach shares all the other general constraints that this guide notes for the Termux command-lines approach. Specifically, to use Mergeall's GUI in Pydroid 3, you'll still need to:

In other words, Mergeall's GUI may run on Android, but it's still running on Android.

Notice the last half of the general requirements listed in this section—it arises because Pydroid 3 shares the exact same permission constraints that we saw earlierfor Termux. Namely, the programs that Pydroid 3 runs can read anywhere on internal storage and removable media, and can write anywhere on internal storage, but writes on removable media are limited to Pydroid 3's app-specific folder only, which is usedlike this, and lives at the following path on your /storage/xxxx-xxxx drive (once you create it, and swap the xs for your drive's Android ID):

Android/data/ru.iiec.pydroid3

It's easy to verify this: see the permissions test script, and its results on both Pydroid 3 and Termux; the results do not differ between Android Nougat and Oreoand are unchanged by installing Pydroid 3's extra-permissions plugin. The takeaway from these results is that storage access rules are the same for both apps, and updates on removable media are limited in both to the app's own folder. The only real difference in the storage-permissions department is that Pydroid 3 replaces Termux's one-time script run with a one-time automatic request on first access to restricted folders.

As we've learned, this means that content on removable media must be nested in Pydroid 3's app-specific folder if it is to be updated by Mergeall in this app. It also means that items to be updated by both Termux and Pydroid 3 must be located in internal storage, because these apps can modify only their own app-specific folders on removable media. Importantly, your choice of Mergeall content folder on SD cards and USB drives wholly determines which access-limited app can change it. For instance, if Mergeall's GUI in Pydroid 3 processes removable-drive content in Termux folders—as in earlier screenshots—it is limited to comparisons only. And while our focus here is on Mergeall, this constraint applies to any GUI run in the app.

This is an unavoidable downside to using Android removable storage in these apps. The lesson here is that _internal storage_—usually at /sdcard—should always be used if possible, because it can be changed by both Termux and Pydroid 3 specifically, and by other apps generally. As an example of the latter set, QPython3, a Python Android app not covered in this guide because it holds no advantages for Mergeall today, has the same storage-access limits per this andthis; Android's rules must be hard to beat.

Update: though uncertain at this writing, Android 11 may impose the same access constraints on internal storage that KitKat imposed on removable media, and invalidate some of this section's coverage in the process. See Appendix C for more on the possible consequences for Mergeall—and every other program that saves shared data on the Android platform.

The Mergeall Pydroid 3 Wrap-Up

So there it is: the good, the bad, and the ugly of using tkinter GUIs like Mergeall's in the Pydroid 3 Android app. To be sure, the tkinter support embedded in this app is nothing short of stunning: it implements nearly all of Tk and tkinter on smartphones, and makes these devices substantially more fun to use, for both programmers and others. You'll have to decide for yourself whether its remaining defects—along with the advertising model of the app that brings it to you—justifies using this support on your phone.

Beyond Pydroid 3's tkinter support, there are additional GUI options for Python programs on Android that we don't have space to cover here. For example, while Termux does not offer tkinter support directly, it does provide some X Windows (a.k.a. X11) GUI support that may provide a path to using tkinter in its Linux environment. For a survey, see its links here andhere, and the X11 searchhere. Even if viable, however, X11 support requires substantial extra setup work that may be too much to ask of users running Mergeall alone.

Other GUI options offer Android-friendlier alternatives, including Kivy,SL4A, and Qt'sPyQt andPySide. Such tools allow programs to use Python for their core logic, but wrap it in GUIs which may better match the Android paradigm. Although their program recoding costs are not trivial, these tools become more attractive for tkinter GUIs that must be redesigned for smaller screens if their audiences are ever to include more than the most-stubborn tkinter fans.

At the end of the day, Pydroid 3's tkinter support is admirable and even amazing, but the approach it offers for Mergeall really just swaps command lines for a GUI run in an IDE that requires extra code changes and comes with noticeable seams. Whether you find it easier to sync your content by running a single command in Termux, or opening and running a source-code file in the Pydroid 3 IDE may very well depend on your background. For this guide's money, though, the latter is just enough extra steps to qualify as runner-up.

Appendix C: Android 11—the Future is Closed?

Preface: This appendix was written when the changes it describes were planned for Android Q. After publication, these changes were pushed out to Android R, and Androids Q and R were rebranded 10 and 11—both respectively and confusingly. Android's names have been updated here, but the rest of this story was left intact; its potentially paralyzing changes still loom large for apps and users in Android 11.

And now for the closing cliff-hanger. True to its thrashing-laden history, Android seems poised to radically alter storage-access permissions again and soon, with potentially profound impacts on both Mergeall and the Android platform in general. Although this change is still tentative, this appendix provides a preview of its implications for users of Android and Mergeall in one possible future.

The Permissions Game Changer

This how-to has gone to great lengths to document the permission rules for internal and removable storage that existed at the time of publication, especially in sections here,here, andhere. As covered, these rules apply to apps running on Androids KitKat through Pie, of which only Oreo and Pie (and later, 10) are actually useable for Mergeall due to an earlier Android timestamps bug.

Regrettably, some of this guide's coverage of the rules for _internal storage_—the unremovable storage space in your phone that much Android documentation misleadingly calls external—might be invalidated in the near future. It now appears that Android 10, released in beta form in March 2019 (when it was known as Q), may impose the same crippling constraints on internal storage that Android KitKat earlier imposed on removable media. In brief, this so-called scoped storage proposal would restrict access to internal storage as follows:

The app sandboxed folder referenced in this list is what we've called the app-specific folder (e.g., in /sdcard/Android) in this document, because its internal-storage version can be accessed by other apps today. Apps already have an app-private folder (e.g., in /data/data), but the new rules would limit apps' access to their own app-specific folder in the currently general internal storage.

This model would be the death knell of same-device interoperability: because files in internal storage would no longer be freely usable across multiple apps, the scope of your on-phone content would be limited to a single role. This is fully unusable for modern computing tasks. By sandboxing apps, the proposed Android severs the data workflow at the heart of content development. It also manages to further trample the ethic of engineering freedom promoted for decades by the open-source world; a platform that imposes tools and rules on this scale would be closed by definition.

Some of the new constraints may be mitigated by wildly proprietary and convoluted interfaces, but these interfaces may be much slower than those of traditional POSIX files, and are unlikely to see widespread use in programs born on less restrictive systems and designed to run on multiple platforms. Either way, this change would impact nearly every program that updates shared content on Android devices (including Mergeall), and could very well be an endgame for this platform's broader potential.

For more background on this proposed change than this appendix can provide, read the terse and perpetually morphing announcement here; the pragmatic technical review by an Android file-explorer developer here; and the outrage of Android developers pleading for a reversal here, here, andhere(the Mergeall team is guilty of the latter).

Caution: the language at some of those links is understandably explicit. If this change becomes permanent—and KitKat's example suggests it will—it threatens to make a host of existing programs and libraries based on the portable POSIX file standard unusable on Android; render Android's internal storage just as locked down and limited as that in iOS; grossly curtail what users can do with their content on their phones; and effectively kill the platform as an option for general-purpose computing.

Simply put, under the scoped-storage model, Android devices may be reduced to launchers of fenced-in apps, and lose all rights to the title of "PC alternative." Indeed, the better analogy for the proposed Android seems the cartridge-based Game Boy of eraspast.

The Impact on Mergeall

That being said, Mergeall itself will likely still work unchanged on Android if these restrictions are imposed, but content on internal storage may have to be structured in the same inconvenient and perilous way as that on removable media. Specifically, it might need to be nested in the updating app's app-specific folder in internal storage. This means the following sandboxed locations for the Termux and Pydroid 3 apps covered in this guide, respectively:

/sdcard/Android/data/com.termux/MY-STUFF /sdcard/Android/data/ru.iiec.pydroid3/MY-STUFF

Unfortunately, content thus stored could generally be updated only by the sole app whose folder hosts its storage, and would be automatically _deleted_if that app is ever uninstalled. While these major constraints already apply to removable media today, extending them to internal storage may make the Mergeall program impractical for many or most users on Android.

There's a slim chance that the more permanent Download shared-collection folder in internal storage would still be usable for Mergeall content in the Termux and Pydroid 3 apps. That is, you may be able to locate your content in internal storage just one level deeper, at:

/sdcard/Download/MY-STUFF

If viable, this might avoid the delete-on-uninstall peril of sandboxed folders, and may support access from multiple apps. Also unfortunately, the beta release's documentationof proposed rules for the Download folder suggests that files stored there may be app specific too, though this is described thinly and ambiguously, and will have to await empirical analysis on widely used devices.

In fact, the very definition of "access" to be restricted is unclear in the beta documents, and also awaits verification in the final release. For example, if the new internal-storage rules shadow those of removable media today, apps may still be able to read anywhere in internal storage, leaving a massive security hole completely unfilled by the new model. But shared-storage _writes_—essential for content creation of any significance—are almost certain to vanish, at least in their current direct form.

As a last resort, the beta also suggests using Android's Storage Access Framework(SAF) to access files created by other apps (e.g., by passing file descriptors down to the native codelayer that hosts Python on Android). But this proprietary library is Java based, unwieldly, and slow; may not work for Mergeall's mass-updates use case; and seems a nonstarter for cross-platform Python code run by an app which sets permissions. Python programs use the POSIX file interface on purpose: it makes them portable to every modern computing platform of interest—except the proposed Android. For much code, the extreme requirements of the SAF would make this platform unsupportable.

In the end, Mergeall and similarly general tools may run on Android 10, but they're likely to be much more difficult and dubious to use, if not altogether pointless. Content collections that are accessible to just one app and may be subject to accidental removal would probably not be worth storing on your phone.

The Impact on Android

Naturally, it's impossible to predict if internal-storage restrictions will turn out to be anything more than beta-release drama, and you'll have to stay tuned for this story's conclusion. There is substantial developer pushback at present; apps like Termux and Pydroid 3 that run Python code may implement broader storage permissions and interfaces in response; phone manufacturers may implement internal storage in ways that cannot be foreseen;and an initially proposed _compatibility mode_promised to retain the current internal-storage permissions of apps used in this guide—at least until those apps targeted later releases, you uninstalled the apps, or you bought a new phone.

What is certain is that deliberately hobbling the Android platform is sure to limit both its users' options, and its devices' scope. Extending proprietary lockdown to internal storage would make portable tools like Mergeall nearly unusable on Android, and constrain the platform to artificially narrow—and even toyish—apps. Such apps might appeal to technically naive masses and generate proportionate advertising revenue in the process, but the closed platform hosting these apps will hold little interest for the skilled development community that produces nontrivial programs used elsewhere.

The net effect would deprive smartphone users of much useful software, and knowingly retard their devices' utility. This would be especially ironic at a time when foldable displays and ample storage hold the promise of expanded roles. Though more subtle and speculative, this change may also help drive smartphone users to cloud-based storage in the face of unworkable on-phone alternatives—boosting subscriptions, trackable content, and single-source control as darker byproducts. In this analysis, Android would remain a carefully crafted arena of exploitation, fueled by greed and justified by red-herring security arguments easily refuted by the platform's own success.

Let's hope that's not what Android's owner has in mind.

Updates: The Rest of the Story

Because Android's internal-storage permissions were still an early and evolving epic when this guide was written, this section has been added to provide updates that arose after publication. Some of its content may be developer oriented, but its twists seem worthy of dramatists of all kinds.

After being amended twice, this updates section grew too unwieldly for inline appearance here, and has been split off to this separate page. Please visit that page for the continuing saga of Python and tkinter on Android 11 and later.

Spoiler: in Android 11+, the All Files Access permission restores general file-path and POSIX access to both shared storage and USB drives, and the Storage Access Framework provides API-based access to single folders. Of these, the former comes at the expense of Play-store presence for some types of apps; the latter is much more an Android-only solution; and both generally require standalone apps. The future may not have turned out to be fully closed, but it did become much more proprietary.