Combined ref pipelines (source+target branch) (#7380) · Issues · GitLab.org / GitLab · GitLab (original) (raw)

Skip to content

Combined ref pipelines (source+target branch)

Problem to Solve

Teams need to keep master "green". It's a huge pain when someone merges a MR that breaks master because they didn't know of some conflict and their MR's pipeline was green. It's great that master has its own pipeline with tests so it won't accidentally deploy a broken codebase, but it's not nice that master is left broken, and any MR based on master will now fail as well.

If we had a way to build the combined result of the merge before merging, it would go a long way towards being more confident in the result.

Solution

In order to keep master green, we can run a pipeline on the state that master would be in after the merge - by combining the source and target branch into a new ref, then running a pipeline on that ref, we can better test that the combined result is also valid (at least moreso than if we test only the source branch.) To achieve this, this issue implements a rule that merge request pipelines now always run by default on the merged result: where the source and target branches are combined into a new ref and a pipeline is run that validates the result prior to merging.

There are some cases where creating a combined ref is not possible or not wanted; in these scenarios (for example, a source branch that has conflicts with the target, or a MR that is still in WIP status) we fallback to a "detached" state and run on the source branch ref - just like it works prior to this issue. Being in this detached state serves as a warning that you are working in a situation where you may eventually have merge problems, and helps highlight that you should get out of WIP status or resolve your merge conflicts as soon as reasonable.

Terminology

Pipelines for merge requests (or merge request pipelines) are pipelines that are created for merge requests. Of these there are two total types:

For clarity, a post-merge pipeline is just the regular pipeline that runs in the target branch after the MR is merged.

Additional Notes

The additional items here are important for implementing this complex feature, but are not required to understand how it will work.

Behavior in pseudo-code:

if MR {
  if !WIP && !conflicts {
    run on merge result ref (merge pipeline - only: merge_request jobs run)
  } else {
    run on branch ref (detached merge pipeline - only: merge_request jobs run)
  }
} else {
  run normal pipeline (not merge pipeline - only: merge_request jobs NOT run)
}

This approach achieves compatibility with current workflows, keeps things as simple as possible, and gives the end user clear control over when to take advantage of this merge request pipeline through use of the WIP status.

User flow

The added functionality of this feature will come into play both when the user pushes new commits and wants to merge.

Commit flow

graph LR

UC>User pushes new commit]
MR{Branch part of MR?}
RBP[Run branch pipeline]
COW{WIP or are there conflicts?}
RMRP[Run detached merge request pipeline]
RMRRP[Run merge request pipeline]
PFSU["Pipeline feedback shown to user (mr pipeline feedback has priority)"]

UC-->MR
MR-->|Yes|COW
MR-->|No|RBP
MR-->|Yes|RBP
COW-->|Yes|RMRP
COW-->|No|RMRRP
RBP-->PFSU
RMRP-->PFSU
RMRRP-->PFSU

Merge flow

graph LR

UM>User wants to merge]
COW{WIP or are there conflicts?}
MUA[Merge button unavailable]
MA[Merge button available]
UPM[User presses merge button]
PTB[Pipeline for target branch is run]
PF{Pipeline failed?}
PS{Pipeline passed?}
PSR{pipeline still running?}
MWPS[Merge when pipeline succeeds button available]
AMP[Starts actual merge into target branch]
HMA{Has source or target advanced?}
PA[Merge process aborted]

UM-->COW
COW-->|Yes|MUA
COW-->|No|PF
PF-->|Yes|MUA
PF-->|No|PSR
PSR-->|Yes|MWPS
PSR-->|No|MA
MWPS-->UPM
UPM-->PS
PS-->|Yes|HMA
PS-->|No|PA
HMA-->|Yes|PA
HMA-->|No|AMP


MA-->UPM
AMP-->PTB

Design and design requirements

FE design specs!

💎 specs

Merge request widget

Relies on the same logic that would be implemented with https://gitlab.com/gitlab-org/gitlab-ce/issues/40246

Requirements Mockups
Merge request is merge-able We will show merge widget as normal (no changes)
Merge request is not merge-able We will show merge widget as normal (no changes)
Merge request is being merged Merge button has spinner and states Merging (no changes)
Fork merge request Warning (note: different from mockup) Fork merge requests do not create merge request pipelines which validate a post merge result Warning merge button (Merge when pipeline succeeds will also be warning colored as we rely on a normal branch pipeline in those cases) fork
Invalid merge request as target-branch advanced (linear history) merge button disabled warning: (note: different from mockup) The target branch has advanced invalidating the merge request pipeline. Please update the source branch and retry merging invalid-prospective-merge-pipeline-target-branch-advanced-linear
Invalid merge request pipeline as target-branch advanced (non-linear history) merge when pipeline succeeds button (which triggers new pipeline when pressed) warning: (note: different from mockup) The target branch has advanced invalidating the merge request pipeline. Please update the source branch and retry merging invalid-prospective-merge-pipeline-target-branch-advanced-non-linear
Merge request pipeline failed ( Note: decision to have no changes for now) merge when pipeline succeeds button (which triggers new pipeline when pressed) warning pipeline failed, retry merge or fix failure prospective-merge-pipeline-failed
Merge request is converted from WIP to non-WIP Merge request pipeline is automatically triggered merge when pipeline succeeds button is shown (no changes)
Merge request widget pipeline block
Requirements Mockups
Normal branch pipeline Unchanged normal_pipeline
Detached merge request pipeline Ref now shows merge request ID/Link before the source branch prospective-merge-pipeline-failed_copy_6
Merged result merge request pipeline Ref now shows merge request ID/Link before source branch and adds target branch prospective-merge-pipeline-failed_copy_7

Pipeline list view

Requirements Mockups
Normal branch pipeline Ref unchanged image
Detached merge request pipeline Ref now shows merge request ID/Link merge request tag is removed Detached tag is added image
Merged result merge request pipeline Ref now shows merge request ID/Link merge request tag is removed image

Pipeline detail view widget

Requirements Mockups
Normal branch pipeline Copy from changed to for normal_pipeline_copy
Detached merge request pipeline Ref now shows merge request ID/Link before the source branch merge request tag is removed Detached tag is added Copy from changed to for normal_pipeline_copy_2
Merged result merge request pipeline Ref now shows merge request ID/Link before source branch and adds target branch merge request tag is removed Copy from changed to for normal_pipeline_copy_3

Job detail view sidebar

Requirements Mockups
Normal branch pipeline Copy from changed to for image
Detached merge request pipeline Ref now shows merge request ID/Link before the source branch Copy from changed to for image
Merged result merge request pipeline Ref now shows merge request ID/Link before source branch and adds target branch Copy from changed to for image

Merge request list view

Requirements Mockups
Pipeline status will include normal pipeline statuses, merge request pipelines statusses, merge request pipeline statuses, and post merge pipeline statusses, which ever one is the latest and thus most important image

Project settings merge request settings

Requirements Mockups
Added option for enabling merge request pipelines for merged result Restructured content (FE only, as same checkboxes still apply) project-settings

What happens on the backend?

To achieve this, the following steps occur in order:

  1. User clicks merge and a merge request pipeline is initialized
  2. The backend requests Gitaly to create new merge request reference already containing the merged codebase (see https://gitlab.com/gitlab-org/gitlab-ce/issues/47110)
  3. GitLab refers to the new, merged .gitlab-ci.yml in the ref to generate the pipeline
  4. Gitlab passes the ref and generated pipeline to the runner to execute (see also gitlab-runner#3983 (closed))
  5. If the pipeline succeeds, and the target branch has not moved ahead, then the source branch is merged into the target. Alternative dismissed approachesThe original MVC here would create a new ref in git that contains the merged codebase we can run against. This approach was discarded due to limitations for handling other common git workflows, such as forking. A branch-only approach is not "upgradable" to forking at a later date, and would require rewriting the feature.

A second proposal was discussed having the runner be responsible for the branch, but due to the possibility that the .gitlab-ci.yml can change in a branch, this does not work - the pipeline is processed by the rails application prior to the runner receiving it.

Usage Measure

We should measure how many people have the feature turned on via the usage ping.

Dogfooding

We (GitLab CE/EE) cannot use this feature in our large projects (i.e., gitlab-ce/gitlab-ee) yet for the following reasons:

In order to adopt it, we need at least the follow up item https://gitlab.com/gitlab-org/gitlab-ce/issues/57581, but likely also https://gitlab.com/gitlab-org/gitlab-ee/issues/9186.

Backward compatibility

By the consequence of modifying Runner and Gitaly for supporting merge ref, users have to update their runners and gitaly to the new version as well as they update rails(CE/EE). If users forgot to update their runner, they will get a message in a pipeline job page that "'Your runner is outdated, please upgrade your runner'" (This is a generic message when runner is too old to support the feature). We already have this mechanizm today thus we don't need additional implementation.

Links / references

Tasks

Common

Data to be persisted

sha = sha(merge-commit)
before_sha = merge_request.diff_base_sha
source_sha = sha(source-branch) = merge_request.diff_head_sha
target_sha = sha(target-branch)

Dataflow

Outscoped tasks

Edited Apr 05, 2019 by Jason Yavorsky