[SR-1639] [SourceKit] Add in-process implementations of sourcekitd functions only defined for XPC

[modocache]

Previous ID	SR-1639
Radar	None
Original Reporter	@modocache
Type	Sub-task
Status	Closed
Resolution	Done

Attachment: Download

Additional Detail from JIRA

Votes	0
Component/s	Package Manager, XCTest
Labels	Sub-task
Assignee	@briancroom
Priority	Medium

md5: e2fa900ee64d58b2f3449789c3d4d373

Parent-Task:

• SR-710 Generate XCTestCaseProvider entries on Linux

blocks:

• SR-1676 Build SourceKit on Linux

Issue Description:

Several SourceKit functions are declared in tools/SourceKit/tools/sourcekitd/include/sourcekitd/sourcekitd.h or tools/SourceKit/tools/sourcekitd/include/sourcekitd/Internal.h, but only implemented in tools/SourceKit/tools/sourcekitd/lib/API/sourcekitdAPI-XPC.cpp. These need implementations that are only included in builds of SourceKit that use an in-process communication model (in other words, not XPC).

Examples of missing implementations include sourcekitd_request_array_create and createErrorRequestInvalid. There are many more.

I noticed these missing implementations when attempting to build SourceKit for Linux (full output available here and as a attachment on this task).

To build SourceKit on Linux yourself, apply swiftlang/swift#2763 and swiftlang/swift#2704 to a branch based off the master branch of apple/swift, then run utils/build-script -R.

Taking the unique symbols from the build output above, I think the following will need "in-process" definitions:

Declared in SourceKit/include/SourceKit/Support/Concurrency.h, implemented for OS X in SourceKit/lib/Support/Concurrency-Mac.cpp:

1. SourceKit::WorkQueue::Impl::create(SourceKit::WorkQueue::Dequeuing, SourceKit::WorkQueue::Priority, llvm::StringRef)

2. SourceKit::WorkQueue::Impl::dispatch(void*, SourceKit::WorkQueue::DispatchData const&)

3. SourceKit::WorkQueue::Impl::dispatchBarrierSync(void*, SourceKit::WorkQueue::DispatchData const&)

4. SourceKit::WorkQueue::Impl::dispatchConcurrent(SourceKit::WorkQueue::Priority, SourceKit::WorkQueue::DispatchData const&)

5. SourceKit::WorkQueue::Impl::dispatchOnMain(SourceKit::WorkQueue::DispatchData const&)

6. SourceKit::WorkQueue::Impl::dispatchSync(void*, SourceKit::WorkQueue::DispatchData const&)

7. SourceKit::WorkQueue::Impl::release(void*)

I think these need implementations for Linux.

Declared in tools/SourceKit/tools/sourcekitd/include/sourcekitd/sourcekitd.h, implemented for OS X in tools/SourceKit/tools/sourcekitd/lib/API/sourcekitdAPI-XPC.cpp:

1. sourcekitd_request_array_create

2. sourcekitd_request_array_set_value

3. sourcekitd_request_dictionary_create

4. sourcekitd_request_dictionary_set_value

5. sourcekitd_request_int64_create

6. sourcekitd_request_release

7. sourcekitd_request_retain

8. sourcekitd_request_string_create

9. sourcekitd_request_uid_create

10. sourcekitd_response_dispose

11. sourcekitd_response_error_get_description

12. sourcekitd_response_error_get_kind

13. sourcekitd_response_get_value

14. sourcekitd_response_is_error

15. sourcekitd_uid_get_from_buf

16. sourcekitd_uid_get_length

17. sourcekitd_uid_get_string_ptr

I think these need implementations for Linux.

Declared in tools/SourceKit/tools/sourcekitd/include/sourcekitd/Internal.h, implemented for OS X in tools/SourceKit/tools/sourcekitd/lib/API/sourcekitdAPI-XPC.cpp:

1. sourcekitd::RequestDict::dictionaryArrayApply(SourceKit::UIdent, llvm::function_ref<bool (sourcekitd::RequestDict)>)

2. sourcekitd::RequestDict::getDictionary(SourceKit::UIdent)

3. sourcekitd::RequestDict::getInt64(SourceKit::UIdent, long&, bool)

4. sourcekitd::RequestDict::getString(SourceKit::UIdent)

5. sourcekitd::RequestDict::getStringArray(SourceKit::UIdent, llvm::SmallVectorImpl<char const*>&, bool)

6. sourcekitd::RequestDict::getUID(SourceKit::UIdent)

7. sourcekitd::RequestDict::getUIDArray(SourceKit::UIdent, llvm::SmallVectorImpl<sourcekitd_uid_s*>&, bool)

8. sourcekitd::ResponseBuilder::Array::appendDictionary()

9. sourcekitd::ResponseBuilder::Dictionary::set(SourceKit::UIdent, SourceKit::UIdent)

10. sourcekitd::ResponseBuilder::Dictionary::set(SourceKit::UIdent, char const*)

11. sourcekitd::ResponseBuilder::Dictionary::set(SourceKit::UIdent, llvm::ArrayRef<llvm::StringRef>)

12. sourcekitd::ResponseBuilder::Dictionary::set(SourceKit::UIdent, llvm::StringRef)

13. sourcekitd::ResponseBuilder::Dictionary::set(SourceKit::UIdent, long)

14. sourcekitd::ResponseBuilder::Dictionary::set(SourceKit::UIdent, sourcekitd_uid_s*)

15. sourcekitd::ResponseBuilder::Dictionary::setArray(SourceKit::UIdent)

16. sourcekitd::ResponseBuilder::Dictionary::setBool(SourceKit::UIdent, bool)

17. sourcekitd::ResponseBuilder::Dictionary::setCustomBuffer(SourceKit::UIdent, sourcekitd::CustomBufferKind, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >)

18. sourcekitd::ResponseBuilder::Dictionary::setDictionary(SourceKit::UIdent)

19. sourcekitd::ResponseBuilder::ResponseBuilder()

20. sourcekitd::ResponseBuilder::createResponse()

21. sourcekitd::ResponseBuilder::getDictionary()

22. sourcekitd::ResponseBuilder::~ResponseBuilder()

23. sourcekitd::createErrorRequestCancelled()

24. sourcekitd::createErrorRequestFailed(char const*)

25. sourcekitd::createErrorRequestInvalid(char const*)

26. sourcekitd::printRequestObject(void*, llvm::raw_ostream&)

I think these need implementations for Linux.

Declared in tools/SourceKit/include/SourceKit/Core/LangSupport.h, implemented for all platforms in tools/SourceKit/lib/SwiftLang/SwiftLangSupport.cpp:

1. SourceKit::LangSupport::SynthesizedUSRSeparator

2. SourceKit::LangSupport::createSwiftLangSupport(SourceKit::Context&)

I think sourcekitdInProc needs to be linked against SourceKitSwiftLang to resolve these symbols.

[modocache]

Do these functions implementations not exist for InProc? Or am I missing something? @akyrtzi, @gribozavr, do you happen to know?

[modocache]

Sent up swiftlang/swift#2766 for the SourceKit::LangSupport symbols.

[modocache]

Sent up swiftlang/swift#2769 for the sourcekitd/sourcekitd.h and sourcekitd/Internal.h symbols. Only the SourceKit::WorkQueue symbols remain.

[modocache]

The SourceKit::WorkQueue symbols are simply wrappers around libdispatch. If we decide to take a dependency upon swift-corelibs-libdispatch, we won't have to provide two implementations of these symbols at all. Not to mention the fact that sourcekitdInProc.cpp already makes use of libdispatch (note the FIXME in that file: "Portability?").

Based on the discussion in swiftlang/swift#2704 however, I have a feeling it might not be that simple to build libdispatch first, then link it to SourceKit, all in a single build script invocation. Advice here would be appreciated!

[modocache]

I searched the mailing lists, but information on building libdispatch is scarce. I sent an email to the mailing lists for advice.

Depending on the state of that project, we may want to avoid depending on a Linux libdispatch build. I took a look at what SourceKit relies on libdispatch for:

• On OS X, it's used to implement SourceKit::WorkQueue from SourceKit/include/SourceKit/Support/Concurrency.h.

• It's used to implement complete-test, an executable used to test SourceKit.

• It's used to implement sourcekitd_send_request and sourcekitd_send_request_sync in tools/sourcekitd/bin/InProc/sourcekitdInProc.cpp.

For each of these, I think we might be able to use asynchronous features from C++ instead (disclaimer: I've only ever written 30 lines of C++).

[modocache]

Sent up swiftlang/swift#2769 to remove usages of libdispatch from tools/sourcekitd/bin/InProc/sourcekitdInProc.cpp.

[briancroom]

Hey Cristian (JIRA User), are you actively working on this? I started taking a stab at some of this before realizing you had assigned this issue to yourself.

[modocache]

I'm pretty sure @briancroom finished this work in swiftlang/swift#3026 Thanks!