[llvm-dev] [monorepo] Downstream branch zipping tool available (original) (raw)

Björn Pettersson A via llvm-dev llvm-dev at lists.llvm.org
Tue Dec 18 02:23:24 PST 2018

Hi David.

Thanks for sharing your branch zipping migration script.

Unfortunately I think our situation is a little bit more complicated.

We have used llvm as the umbrella repo, so in llvm we have a "master" branch (from the git single repo version of llvm) and a couple of downstream branches (let's call them "down0", "down1") containing our downstream work (with frequent merges from "master"). The downstream branches has tools/clang and runtimes/compiler-rt as submodules, as well as a couple of downstream submodules.

In our downstream version of clang we have a similar structure. A "master" branch (mapping to the git single repo version clang), and a couple of downstream branches. The downstream branches has tools/extra (i.e. clang-tools-extra) as a submodule.

I can also mention that the clang, compiler-rt and clang-tools-extra submodules aren't present from the beginning of history. They have been added later on.

I doubt that zip-downstream-fork.py will work out-of-the-box. Hopefully I'll be able to patch it for our scenario. Any guidelines might be helpful. But maybe it isn't even worth trying to adapt zip-downstream-fork.py to do something useful for our scenario?

If someone else got a similar scenario, let me know. Perhaps we can do some joint effort in adapting the zipper script.

Regards, Björn

-----Original Message----- From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of David Greene via llvm-dev Sent: den 12 november 2018 22:27 To: llvm-dev at lists.llvm.org; cfe-dev at lists.llvm.org; libcxx- dev at lists.llvm.org; lldb-dev at lists.llvm.org; openmp-dev at lists.llvm.org; libclc-dev at lists.llvm.org; clangd-dev at lists.llvm.org Subject: [llvm-dev] [monorepo] Downstream branch zipping tool available

Building on the great work that James Knight did on migrate-downstream-fork.py (Thanks, James!) [1], I've created a simple tool to take migrated downstream fork branches and zip them into a single history given a history containing submodule updates of subprojects [2]. With migrate-downstream-fork.py, one is left with a set of unrelated histories, one per subproject: llvm clang compiler-rt * V Add my fancy LLVM feature * G Fix my dumb clang bug * Z Merge from upstream compiler-rt One can do an octopus merge to unify them: *-- Merge llvm, clang and compiler-rt _|\ _ * \ \ V Add my fancy LLVM feature | * | G Fix my dumb clang bug | | * Z Merge from upstream compiler-rt Unfortunately, that doesn't show the logical history of development, where changes were effectively applied to subprojects in a linear fashion. This makes it more difficult to do bisects, among other things because none of the downstream integration happens until the octopus merge. Let's say that downstream you have a local mirror for each LLVM subproject you work on. Suppose also that you have an "umbrella" repository that holds submodule references to all those local mirrors. Various commits in the umbrella update submodule references: * Update llvm submodule to V * Update clang submodule to G * Don't update any submodules, fix scripts or something * Update compiler-rt submodule to Z | zip-downstream-fork.py will take these submodule updates and "inline" them into the umbrella history, making it appear that the downstream commits were applied against the monorepo in the order implied by the umbrella history: * A Add my fancy LLVM feature * B Fix my dumb clang bug * C Merge from upstream compiler-rt | Parent relationships for merges from upstream are preserved, though as top-level comments in zip-downstream-fork.py explain, the history graph can look a little strange. Commits that don't update submodules are skipped on the assumption that they modify things uninteresting to a monorepo history. Such commits could be preserved but doing so has some caveats as explained in the comments. Perhaps your umbrella repository holds your build scripts. You'd probably want to migrate that to the zipped history. If there's strong demand for this I could look into doing it. There are various other limitations to the tool explained in the comments. It was enough to get us going and I'm hopeful it will be useful for others. It seems to do the right thing with our repositories but YMMV. Feel free to open PRs with bug fixes. :) To get this to work, you'll need to apply a PR for migrate-downstream-fork.py to fix issues with --revmap-out [3]. -David [1] https://github.com/jyknight/llvm-git-migration/blob/master/migrate- downstream-fork.py [2] https://github.com/jyknight/llvm-git- migration/pull/2/commits/a3b44a294c20f1762cb42b5794e6130c5b27f22d [3] https://github.com/jyknight/llvm-git-migration/pull/1

LLVM Developers mailing list llvm-dev at lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

More information about the llvm-dev mailing list