Extending functionality of Build Server Protocol with SourceKit-LSP (original) (raw)

September 5, 2024, 8:43pm 1

Extending functionality of Build Server Protocol with SourceKit-LSP

SourceKit-LSP has been able to connect to a build server through the Build Server Protocol (BSP) to get build settings for a source file and thus provide intelligent editor functionality for a few years. Recently, SourceKit-LSP has gained a deeper understanding of a project’s structure (in particular targets and target dependencies) to support background indexing in SwiftPM projects. We would like to open this capability up to other build systems by extending SourceKit-LSP’s interaction through BSP, building on top of SourceKit-LSP for other build systems - #12 by blangmuir with new knowledge that we gained by implementing background indexing for SwiftPM projects.

The goal is that BSP is used for all interactions between SourceKit-LSP and build systems, including the built-in build systems (SwiftPM and compile_commands.json) so that all functionality is guaranteed to be exposed to external build systems.

As we implement the integration with BSP and learn more about the interaction, the exact details might change and this post might become outdated. We plan to have some reference documentation in SourceKit-LSP that describes the necessary requests that need to be implemented by a BSP server for SourceKit-LSP, which should be kept up-to-date.

Current BSP Integration

For background, the current interaction between SourceKit-LSP and a BSP server works as follows:

The only standard BSP request that is currently in use build/initialize.

Proposed new BSP Integration

BSP already has requests that can represent the structure of targets in the project and we are planning to re-use those for SourceKit-LSP. BSP servers will still need to implement some custom requests to support SourceKit-LSP, such as a request to get build settings. The following is split into 4 sections:

Standard BSP requests

Note that build systems that don’t have a notion of targets (eg. you can consider compile_commands.json as such a build system), it is acceptable that they only provide a single dummy target for all files.

BSP Extensions required for SourceKit-LSP functionality

textDocument/sourceKitOptions

/** Request from SourceKit-LSP to the BSP server to retrieve the compiler 
  * arguments that should be used to build the given file in the given target */
export interface BuildSettingsParams {
  textDocument: TextDocumentIdentifier;
  target: BuildTargetIdentifier;
}

export interface BuildSettingsResponse {
  /** The arguments that should be passed to a compiler to build this file */
  compilerArguments: string[];
  
  /** The working directory in which the compiler should be run. 
   * If `null`, the file should be built without a working directory. */
  workingDirectory?: string;
}

This is the key request that BSP servers need to implement for SourceKit-LSP. It provides the compiler arguments that are used to provide semantic functionality for the file. This was originally proposed in SourceKit-LSP for other build systems - #12 by blangmuir but never implemented.

Closest BSP equivalent: buildTarget/cppOptions in the C++ extension of BSP but its structure is quite different because it eg. explicitly specifies defines aka. -D flags. textDocument/sourceKitOptions is more opaque.

Caching: The result of this request is assumed to be stable and cacheable until the BSP server sends a buildTarget/didChange for the target passed in this request.

As described above, the current BSP integration pushes build settings from the build server to the client. SourceKit-LSP has been redesigned to be more pull-based. This pull-based model fits better to recent developments in LSP where eg. diagnostics also became a pull request, initiated by the editor, instead of the LSP server pushing diagnostics. We would thus like to deprecate the existing build/sourceKitOptionsChanged notification from the BSP server to the client in favor of this pull model.

BSP Extensions required for Background Indexing

buildTarget/prepare

/** Build Swift modules for this target so that source files in downstream 
 * modules can import modules defined by this target */
export interface PrepareParams {
  /** A sequence of build targets to compile. */
  targets: BuildTargetIdentifier[];
  /** A unique identifier generated by the client to identify this request.
   * The server may include this id in triggered notifications or responses. */
  originId?: OriginId;
}

export interface PrepareResult {}

The closest BSP equivalent is buildTarget/compile but we have decided to use a different method because preparation doesn’t need to generate any binaries and the generated Swift modules only need to contain declarations, no bodies. It is thus different from a compile.

In the future, we could consider running buildTarget/compile if the build server doesn’t support buildTarget/prepare in order to try preparing a module. The problem is that a normal compile usually doesn’t continue building if a target’s dependency fails to build but preparation is expected to continue here.

workspace/waitForBuildSystemUpdates

This is a no-op. If the build system is currently processing updates, like file changes or is currently building the build graph, it should only return once they have finished processing. This is used by background indexing for eg. the current scenario:

BSP Extensions to help BSP Servers

workspace/didChangeWatchedFiles

Correctly watching for file changes on all platforms is non-trivial. To help BSP servers that are developed with SourceKit-LSP as its primary user, SourceKit-LSP can forward the LSP notification workspace/didChangeWatchedFiles from the editor to the BSP server. To receive the file change notifications, the BSP server must set the watches: FileSystemWatcher[] parameter in the initialize response, specifying the glob patterns to watch for.

dima_kozhinov (Dmitry Kozhinov) September 5, 2024, 9:03pm 2

Sounds interesting. I'm not familiar with BSP, but I certainly interested in the possibility of SourceKit-LSP supporting alternative build systems besides SwiftPM.

rockbruno (Bruno Rocha) September 9, 2024, 7:58am 3

This is great! I have been recently looking into expanding sourcekit-lsp to work with Bazel projects, and it seems this is exactly what I would need in this case.

jaredh159 (Jared Henderson) September 12, 2024, 4:50pm 4

very happy to hear about ongoing work on SourceKit-LSP. thank you!

i use neovim as my daily driver, will these changes improve the problem of sourcekit not understanding Xcode projects? i currently architect all my projects to have almost all my code in SPM packages and only thin entrypoints calling out to SPM modules within Xcode projects, because that way i can use neovim with the LSP.

skortekaas (Sam Kortekaas) September 25, 2024, 2:25pm 7

Great to see a proposal for this! I'm mostly curious to get more details on how the buildTarget/prepare request will work and how background indexing will be handled in this mode. I understand that you probably don't have the details yet, but I have some high-level questions that come to mind:

Some minor comments:

I suggest posting this proposal on the BSP Discussions page to gather some feedback from the BSP community.

ahoppen (Alex Hoppen) October 2, 2024, 6:45pm 8

SourceKit-LSP will call into the build system to prepare a target and then handle the reindexing itself. sourcekit-lsp/Contributor Documentation/Background Indexing.md at main · swiftlang/sourcekit-lsp · GitHub contains some information about how indexing works. It might not answer your question directly but probably contains some information you might find interesting.

buildTarget/prepare should not update the index store. SourceKit-LSP will update the index store when it indexes the source files after preparation is done. Background indexing will populate the index store path that was returned by the build/initialize response.

A recent iteration added support for window/workDoneProgress/create and $/progress to show progress in the editor. Does that satisfy your needs?

Do you happen to have a link to the Metals’ BSP extensions and how it communicates file changes? I couldn’t find any.

Also, the sourcekit-lsp repo now contains documentation for the BSP Extensions used by SourceKit-LSP and a basic guide of the steps needed to implement a BSP server. If you have feedback on the integration, I would like to hear about it.

skortekaas (Sam Kortekaas) October 25, 2024, 1:51pm 9

Thanks for the clarifications. I read the document you added and it looks good to me. One thing I'm not sure about is the use of window/workDoneProgress/create and $/progress. I think the server should notify the client using build/taskStart, build/taskProgress, build/taskFinish. Where the type of tasks that can be reported is not limited to build-related tasks (see Build Server Protocol | Build Server Protocol). The LSP can then propagate these notifications to the editor using window/workDoneProgress/create and $/progress. See scala.meta.internal.metals.clients.language.ForwardingMetalsBuildClient in Scala Metals for an example.

Regarding file watching in Metals, I think I might have misunderstood their approach. I browsed the code and it looks like the client (Metals LSP) doesn't directly send file change notifications to the server. It merely invokes compilation for affected targets when scala.meta.internal.metals.MetalsLspService#didChangeWatchedFiles is invoked in the client. I think the approach you took with workspace/didChangeWatchedFiles is better.

ahoppen (Alex Hoppen) October 25, 2024, 6:52pm 10

Using build/taskStart etc. for status reporting instead of the BSP extensions for window/workDoneProgress/create and $/progress does seem like a good idea. I’m not sure how I missed them. I opened Use `build/taskStart`, `build/taskProgress` and `build/taskFinish` to communicate progress from a BSP server to the client · Issue #1783 · swiftlang/sourcekit-lsp · GitHub to switch that, thanks for the suggestion!

buttler (Josh Buttler) November 8, 2024, 5:28am 12

Thankyou for giving solution. Your answer is worked.