Rebuilding the entire Debian archive twice on arm64 hardware for fun and proft (original) (raw)

[ Please note the cross-post and respect the Reply-To... ]

Hi folks,

This has taken a while in coming, for which I apologise. There's a lot of work involved in rebuilding the whole Debian archive, and many many hours spent analysing the results. You learn quite a lot, too! :-)

I promised way back before DC18 that I'd publish the results of the rebuilds that I'd just started. Here they are, after a few false starts. I've been rebuilding the archive specifically to check if we would have any problems building our 32-bit Arm ports (armel and armhf) using 64-bit arm64 hardware. I might have found other issues too, but that was my goal.

The logs for all my builds are online at

https://www.einval.com/debian/arm/rebuild-logs/

for reference. See in particular

https://www.einval.com/debian/arm/rebuild-logs/armel/FAIL/FAIL.html https://www.einval.com/debian/arm/rebuild-logs/armhf/FAIL/FAIL.html

for automated analysis of the build logs that I've used as the basis for the stats below.

Executive summary

As far as I can see we're basically fine to use arm64 hosts for building armel and armhf, so long as those hosts include hardware support for the 32-bit A32 instruction set. As I've mentioned before [1] that's not a given on all arm64 machines, but there are sufficient machine types available that I think we should be fine. There are a couple of things we need to do in terms of setup - see "Machine configuration" below.

[1] https://lists.debian.org/debian-arm/2018/06/msg00062.html

Methodology

I (naïvely) just attempted to rebuild all the source packages in unstable main, at first using pbuilder to control the build process and then later using sbuild instead. I didn't think to check on the stated architectures listed for the source packages, which was a mistake - I would do it differently if redoing this test. That will have contributed quite a large number of failures in the stats below, but I believe I have accounted for them in my analysis.

I built lots of packages, using a range of machines in a small build farm at home:

using my local mirror for improved performance when fetching build-deps etc. I started off with a fixed list of packages that were in unstable when I started each rebuild, for the sake of simplicity. That's one reason why I have two different numbers of source packages attempted for each arch below. If packages failed due to no longer being available, I simply requeued using the latest version in unstable at that point.

I then developed a script to scan the logs of failed builds to pick up on patterns that matched with obvious causes. Once that was done, I worked through all the failures to (a) verify those patterns, and (b) identify any other failures. I've classified many of the failures to make sense of the results. I've also scanned the BTS for existing bugs matching my failed builds (and linked to them), or filed new bugs where I could not find matches.

I did not investigate fully every build failure. For example, where a package has never been built before on armel or armhf and failed here I simply noted that fact. Many of those are probably real bugs, but beyond the scope of my testing.

For reference, all my scripts and config are in git at

https://git.einval.com/cgi-bin/gitweb.cgi?p=buildd-scripts.git

armel results

Total source packages attempted: 28457 Successfully built: 25827 Failed: 2630

Almost half of the failed builds were simply due to the lack of a single desired build dependency (nodejs:armel, 1289). There were a smattering of other notable causes:

The stats that matter are the arch-specific failures for armel:

and the new bugs I filed:

Considering the number of package builds here, I think these numbers are basically noise. The vast majority of the failures I found were either already known in the BTS (260), unrelated to what I was looking for, or both.

See below for more details about host configuration for armel builds.

armhf results

Total source packages attempted: 28056 Successfully built: 26772 Failed: 1284

FTAOD: I attempted fewer package builds for armhf as we just had a smaller number of packages when I started that rebuild. A few weeks later, it seems we had a few hundred more source packages for the armel rebuild.

The armhf rebuild showed broadly the same percentage of failures, if you take into account the nodejs difference - it exists in the armhf archive, so many hundreds more packages could build using it.

In a similar vein for notable failures:

Arch-specific failures found for armhf:

and the new bugs I filed:

• 1 bugs for arch misdetection • 8 bugs for alignment problems • 10 bugs for arch-specific test failures • 3 bugs for arch-specific misc failures

Again, these small numbers tell me that we're fine. I liked to 139 existing bugs in the BTS here.

Machine configuration

To be able to support 32-bit builds on arm64 hardware, there are a few specific hardware support issues to consider:

Bug highlights

Thanks

I've spent a lot of time looking at existing FTBFS bugs over the last weeks, to compare results against what I've been seeing. Much kudos to people who have been finding and filing those bugs ahead of me, in particular Adrian Bunk and Matthias Klose who have filed many such bugs. Also thanks to Helmut Grohne for his script to pull down a summary of FTBFS bugs from UDD - that saved many hours of effort!

Please let me know if you think you've found a problem in what I've done, or how I've analysed the results here. I still have my machines set up for easy rebuilds, so reproducing things and testing fixes is quite easy - just ask!

-- Steve McIntyre, Cambridge, UK. steve@einval.com Armed with "Valor": "Centurion" represents quality of Discipline, Honor, Integrity and Loyalty. Now you don't have to be a Caesar to concord the digital world while feeling safe and proud.

Attachment:signature.asc
Description: PGP signature

Reply to: