Support for resolving private packages through HTTPS with xcodebuild (original) (raw)

Hi ,
We are using the Swift Package Manager in some of our Xcode projects to integrate packages that live in other private repos of the organization. Everything works fine, except when we run builds on CI. Despite having the right Git configuration to authenticate successfully with our Git provider, the xcodebuild process fails with the following error:

xcodebuild: error: Could not resolve package dependencies:
  Failed to clone repository https://github.com/foo/bar:
    Cloning into bare repository '/Users/anka/Library/Developer/Xcode/DerivedData/...'...
    fatal: could not read Username for 'https://github.com': terminal prompts disabled

After some reading, I found this blog post that mentions that HTTPS is not supported, and also noticed that Apple's official documentation mentions suggests to use SSH. Unfortunately, using SSH is not possible with GitHub apps, and the proxy solution suggested in the former post seems too much overhead.

Is there a way for xcodebuild's package resolution to successfully do authentication through HTTPS using the global git configuration?

rlovelett (Ryan Lovelett) June 9, 2021, 12:58pm 2

We use HTTPS and SSH at my work interchangeably using both Xcode and xcodebuild (and vanilla SwiftPM for that matter) in our CI. In the end, I think it is fair to say that package manager is not the limitation here but likely your ability to supply the correct credentials while running in your CI.

How are you providing your credentials for https://github.com during your CI runs?

NeoNacho (Boris Buegling) June 9, 2021, 6:16pm 3

If you use xcodebuild -scmProvider xcode, HTTPS can be used, but you would typically log in via Xcode preferences to your SCM service account.

Let me try to find out whether there is a way to accomplish this on a CI system as well.

NeoNacho (Boris Buegling) June 9, 2021, 8:43pm 4

Looks like there is no alternative to using the Xcode UI today, so you would need to login once via that on the CI machine to store the credentials and then using xcodebuild -scmProvider xcode should be able to use those credentials you configured.

pepi (Pedro Piñera) June 14, 2021, 7:12pm 5

Hi!
Thanks for your answer. Adding a bit of more context, the way we authenticate on CI hosts is by using git's GIT_ASKPASS environment variable, and passing an executable that tells Git where to read the username and password from. That's working fine when we run Git in the system.

@NeoNacho do you know if we can automate the process of storing the credentials where Xcode expects them? Unfortunately, setting the token in our CI hosts through Xcode's UI is not viable because we rotate the credentials periodically. Thanks in advance!

NeoNacho (Boris Buegling) June 15, 2021, 5:38pm 6

Oh, if git on the system works fine, -scmProvider system is actually the right thing. This is the default in Xcode 12.5 and later, perhaps you are using a previous version?

pepi (Pedro Piñera) June 17, 2021, 1:52pm 7

Interesting, we are for instance using Xcode 12.5. Is it possible that the GIT_ASKPASS variable is not exposed to the Swift Package Manager process? We've verified that the variable works at the system level because we can clone private repositories. I don't know what else could be .

rlovelett (Ryan Lovelett) June 17, 2021, 4:03pm 8

The way we do it in our CI is "this one weird trick". We (ab)use Git's url..insteadOf to re-write the clone URL to include the appropriate credentials in our pipeline and make use of xcodebuild's -usePackageSupportBuiltinSCM.

Our CI of choice is GitLab and it provides ephemeral credentials as environment variables to clone over HTTPS for each job. Specifically it provides, CI_JOB_TOKEN, CI_SERVER_PROTOCOL, and CI_SERVER_HOST.

Typically, our URLs in our Xcode and Package.swift files are of the SSH variety (e.g., git@my.git.server.org:group/project.git). So we use Git's url..insteadOf to re-write the URL to be HTTP(S) and to include the appropriate credentials.

Example

git config --global url.$CI_SERVER_PROTOCOL://gitlab-ci-token:$CI_JOB_TOKEN@$CI_SERVER_HOST/.insteadOf git@$CI_SERVER_HOST:

Assuming that CI_SERVER_HOST=my.git.server.org, CI_SERVER_PROTOCOL=https, and CI_JOB_TOKEN=WF0IjoxNTE2MjM5MDIyfQ.

This translates any URL in either or Xcode or Package.swift that might look like git@my.git.server.org:group/project.git → https://gitlab-ci-token:WF0IjoxNTE2MjM5MDIyfQ@my.git.server.org/group/project.git.

Then using xcodebuild -usePackageSupportBuiltinSCM along with all of our other normal flags things build just fine.

That git config is cut-n-paste from our .gitlab-ci.yml file and we have found this to be a relatively painless and robust way of handling HTTPS Git authentication. YMMV.

Hope it helps.

pepi (Pedro Piñera) June 18, 2021, 10:36am 9

Thanks @rlovelett, we were not passing the -usePackageSupportBuiltinSCM argument when running xcodebuild. I added the argument, but it still fails. This is the entire xcodebuild command that we run.

xcodebuild build-for-testing -workspace Appxcworkspace -configuration Debug -scheme Tests -scmProvider system -usePackageSupportBuiltinSCM -destination '...' 

One thing to note, is that we don't authenticate using the GIT_ASKPASS variable as I mentioned, but we create a gitconfig file in the system, that has a credentials helper set that takes care of passing the username and password when Git needs them. The reason why we don't put a token there is because we rotate those for security reasons, and we need the Git process to get it from another process. Looking at the logs:

xcodebuild: error: Could not resolve package dependencies:
  Failed to clone repository https://github.com/organization/repo:
    Cloning into bare repository '/Users/anka/Library/Developer/Xcode/DerivedData/Shopify-eotrbxjhnrnkesdptcwwzebeoowa/SourcePackages/repositories/repo-0dd4e281'...
    fatal: could not read Username for 'https://github.com': terminal prompts disabled

I wonder if the Git process fails because we disable the terminal prompts in the process run by Xcode? Shouldn't Git read the credential helper from the gitconfig file, and shell out to the helper instead of trying to read it from the standard input?

rlovelett (Ryan Lovelett) June 18, 2021, 9:35pm 10

This is more or less what GitLab does too. Ours rotate for every job, hence the environment variable. If it were my build environment I'd switch to using environment variables. But this is not strictly necessary I suppose.

How? Are you sure the .gitconfig file you are creating the right .gitconfig file? We set our .gitconfig properties via git config --global rather than writing the file ourselves. Ensuring that git should always know the correct location.

pepi (Pedro Piñera) June 21, 2021, 10:57am 11

We create a file /usr/local/etc/gitconfig containing among other things:

[credential "https://github.com"]
  helper = !path/to/binary git credentials
  useHttpPath = true

With that configuration, we can clone private repositories successfully. The problem seems to be Xcode not picking that configuration for some reason.

tarbayev (Nickolay Tarbayev) January 16, 2023, 6:11pm 12

The most reliable solution for me was using netrc file:

echo -e "machine github.com\n  login oauth2\n  password $GITHUB_TOKEN\n\n" > ~/.netrc

It works well without additional setup for both xcodebuild (even without -scmProvider system) and swift build and does not create any records in the keychain (which might fail other jobs).

menjoo (Menno Morsink) February 8, 2024, 3:17pm 13

For us using the netrc file does the trick. But note that besides:

machine github.com
  login <username>
  password <personal access token>

You also need to add this if your Package.swift points to binaries stored in GitHub package registry:

machine maven.pkg.github.com
  login <username>
  password <personal access token>

Took me quite a while to figure this out, but sub domains are not included in the machine definitions, so specify them individually.