[Windows] Support Nightly Toolchains for main and release/5.10

[shahmishal]

Windows toolchain needs to be built on ci-external.swift.org for main and release/5.10 branch.

[shahmishal]

Currently blocked on #69483

[shahmishal]

cc: @grynspan @compnerd

[shahmishal]

Currently blocked on getting this PR merged #69555

[shahmishal]

Job created for main branch: https://ci-external.swift.org/job/swift-main-windows-toolchain/20/

[shahmishal]

Job failed to find zlib

CMake Error: The source directory "C:/Users/swift-ci/jenkins/workspace/swift-main-windows-toolchain/zlib" does not exist.

Found the issue, and started new build: https://ci-external.swift.org/job/swift-main-windows-toolchain/21/console

[shahmishal]

@compnerd We are seeing following test failing, it might be config issue. However, if you can look into this would be helpful. Thanks!

https://ci-external.swift.org/job/swift-main-windows-toolchain/21/console

Failed Tests (65):
  Swift(windows-x86_64) :: Driver/android-link.swift
  Swift(windows-x86_64) :: Driver/driver_mode.swift
  Swift(windows-x86_64) :: Driver/help.swift
  Swift(windows-x86_64) :: Driver/options-repl.swift
  Swift(windows-x86_64) :: Driver/print_target_info.swift
  Swift(windows-x86_64) :: Driver/sdk.swift
  Swift(windows-x86_64) :: Driver/working-directory.swift
  Swift(windows-x86_64) :: IDE/complete_macros_expanded.swift
  Swift(windows-x86_64) :: IDE/complete_optionset.swift
  Swift(windows-x86_64) :: Index/index_macros.swift
  Swift(windows-x86_64) :: Interop/CxxToSwiftToCxx/bridge-cxx-struct-back-to-cxx-execution.cpp
  Swift(windows-x86_64) :: Interop/SwiftToCxx/stdlib/swift-stdlib-in-cxx.swift
  Swift(windows-x86_64) :: Macros/accessor_macros.swift
  Swift(windows-x86_64) :: Macros/attached_macros_diags.swift
  Swift(windows-x86_64) :: Macros/composed_macro.swift
  Swift(windows-x86_64) :: Macros/extension_macro_plugin.swift
  Swift(windows-x86_64) :: Macros/if_expr.swift
  Swift(windows-x86_64) :: Macros/imported_conformance_macro.swift
  Swift(windows-x86_64) :: Macros/init_accessor.swift
  Swift(windows-x86_64) :: Macros/lazy_parsing.swift
  Swift(windows-x86_64) :: Macros/macro_and_typealias.swift
  Swift(windows-x86_64) :: Macros/macro_expand.swift
  Swift(windows-x86_64) :: Macros/macro_expand_attributes.swift
  Swift(windows-x86_64) :: Macros/macro_expand_closure.swift
  Swift(windows-x86_64) :: Macros/macro_expand_codeitems.swift
  Swift(windows-x86_64) :: Macros/macro_expand_conformances.swift
  Swift(windows-x86_64) :: Macros/macro_expand_conformances_multi.swift
  Swift(windows-x86_64) :: Macros/macro_expand_extensions.swift
  Swift(windows-x86_64) :: Macros/macro_expand_member_with_conformances.swift
  Swift(windows-x86_64) :: Macros/macro_expand_other.swift
  Swift(windows-x86_64) :: Macros/macro_expand_peers.swift
  Swift(windows-x86_64) :: Macros/macro_expand_primary.swift
  Swift(windows-x86_64) :: Macros/macro_expand_synthesized_members.swift
  Swift(windows-x86_64) :: Macros/macro_expand_throwing.swift
  Swift(windows-x86_64) :: Macros/macro_expand_variadic.swift
  Swift(windows-x86_64) :: Macros/macro_fixits.swift
  Swift(windows-x86_64) :: Macros/macro_invalid.swift
  Swift(windows-x86_64) :: Macros/macro_keywordname.swift
  Swift(windows-x86_64) :: Macros/macro_mixedpluginpath.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_basic.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_error.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_searchorder.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_server.swift
  Swift(windows-x86_64) :: Macros/macro_shadowing.swift
  Swift(windows-x86_64) :: Macros/macro_swiftdeps.swift
  Swift(windows-x86_64) :: Macros/macro_vfs.swift
  Swift(windows-x86_64) :: Macros/magic_literal_expansion.swift
  Swift(windows-x86_64) :: Macros/top_level_freestanding.swift
  Swift(windows-x86_64) :: ModuleInterface/Observable.swift
  Swift(windows-x86_64) :: ModuleInterface/macros.swift
  Swift(windows-x86_64) :: Profiler/coverage_macros.swift
  Swift(windows-x86_64) :: SIL/Serialization/deserialize_stdlib.sil
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-clang-implicit.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-forwarding-module.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-typecheck-module-from-interface.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map.swift
  Swift(windows-x86_64) :: Serialization/macros.swift
  Swift(windows-x86_64) :: SourceKit/Macros/diags.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_across_modules.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_formatting.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_semantic_token.swift
  Swift(windows-x86_64) :: SourceKit/Macros/syntactic_expansion.swift
  Swift(windows-x86_64) :: SymbolGraph/Symbols/Macro.swift
  Swift(windows-x86_64) :: SymbolGraph/Symbols/Mixins/DeclarationFragments/Full/Macros.swift
  Swift(windows-x86_64) :: stdlib/llvm-support-odr-violation.test-sh


Testing Time: 759.15s
  Skipped          :    1
  Unsupported      : 2459
  Passed           : 7591
  Expectedly Failed:   47
  Failed           :   65

[compnerd]

@shahmishal super exciting!

Just looking through a couple of things stand out:

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\vsdevcmd.bat" -arch=amd64 -host_arch=amd64
'C:\Program' is not recognized as an internal or external command,
operable program or batch file.

With the switch to build.ps1, this shouldn't be needed (build.ps1 will properly setup the VS Dev environment to support building the SDK for all the architectures). Something that might help simplify the Jenkins configuration.

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>d /s /q C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\build
'd' is not recognized as an internal or external command,
operable program or batch file.

Odd, I'm not sure where this is coming from. We should investigate this later.

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>powershell.exe -ExecutionPolicy RemoteSigned -File C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\utils\build.ps1 -SourceCache C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain -BinaryCache T: -ImageRoot T: -BuildType Release -Test swift,dispatch,foundation,xctest, -Stage T:\artifacts

This looks right ... and if this is failing, I doubt it's a configuration issue and more likely this is an issue in build.ps1. I'll need a little bit to dig into this a bit further to try to understand what is going on.

[compnerd]

There seem to be a few failures due to %swift_driver_plain being expanded to main. This would mean that config.swift = inferSwiftBinary('swift') returned main? I don't see that in the log, so I expect that it found it in the path.

[shahmishal]

If it helps, here is the build step:

            "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\vsdevcmd.bat" -arch=amd64 -host_arch=amd64"

            rd /s /q %LocalAppData%\\clang\\ModuleCache


            set CMAKE_BUILD_TYPE=Release
            set PYTHON_HOME="%ProgramFiles(x86)%\\Microsoft Visual Studio\\Shared\\Python37_64"

            set SourceRoot=%WORKSPACE%
            set BuildRoot=%SourceRoot%\\build

            md %BuildRoot%
            subst T: /d
            subst T: %BuildRoot% || (exit /b)

            set BuildRoot=T:

            set PackageRoot=%BuildRoot%\\artifacts

            md %PackageRoot%

            :: Setup temporary directories
            md %BuildRoot%\\tmp
            set TEMP=%BuildRoot%\\tmp
            set TMP=%BuildRoot%\\tmp
            set TMPDIR=%BuildRoot%\\tmp

            set NINJA_STATUS=[%%f/%%t][%%p][%%es]

            set TestArg=-Test swift,dispatch,foundation,xctest,

            set "SkipPackagingArg= "

            rd /s /q %WORKSPACE%\\build

            powershell.exe -ExecutionPolicy RemoteSigned -File %WORKSPACE%\\swift\\utils\\build.ps1 ^
            -SourceCache %SourceRoot% ^
            -BinaryCache %BuildRoot% ^
            -ImageRoot %BuildRoot% ^
            -BuildType %CMAKE_BUILD_TYPE% ^
            %SkipPackagingArg% ^
            %TestArg% ^
            -Stage %PackageRoot%

[shahmishal]

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>d /s /q C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\build
'd' is not recognized as an internal or external command,
operable program or batch file.

Odd, I'm not sure where this is coming from. We should investigate this later.

I have a fix for this.

[compnerd]

It appears that a number of the failures are referencing ../lib/swift/windows and not being able to find sources under there. I think that this might be due to the split of the build tree and the source tree.

I wonder if there was something that we had done before that results in the build tree being linked or something?

[compnerd]

Hmm, I wonder if it is the PYTHON_HOME that is causing the issue ... we use python 3.9, but PYTHON_HOME is set to 3.7. Would it be possible to test with PYTHON_HOME unset?

[compnerd]

Bad Run:

lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:210: note: Using cmake: C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/bin/cmake.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-frontend: t:\1\bin\swift-frontend.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swiftc: t:\1\bin\swiftc.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-opt: t:\1\bin\sil-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-func-extractor: t:\1\bin\sil-func-extractor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-llvm-gen: t:\1\bin\sil-llvm-gen.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-nm: t:\1\bin\sil-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-passpipeline-dumper: t:\1\bin\sil-passpipeline-dumper.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using lldb-moduleimport-test: t:\1\bin\lldb-moduleimport-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-ide-test: t:\1\bin\swift-ide-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-dependency-tool: t:\1\bin\swift-dependency-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-reflection-dump: t:\1\bin\swift-reflection-dump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-remoteast-test: t:\1\bin\swift-remoteast-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-indent: t:\1\bin\swift-indent.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-symbolgraph-extract: t:\1\bin\swift-symbolgraph-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-api-extract: t:\1\bin\swift-api-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using clang: t:\1\bin\clang.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using clang++: t:\1\bin\clang++.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-link: t:\1\bin\llvm-link.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-llvm-opt: t:\1\bin\swift-llvm-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-profdata: t:\1\bin\llvm-profdata.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-cov: t:\1\bin\llvm-cov.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-strings: t:\1\bin\llvm-strings.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using FileCheck: t:\1\bin\filecheck.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-dwarfdump: t:\1\bin\llvm-dwarfdump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-readelf: t:\1\bin\llvm-readelf.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-dis: t:\1\bin\llvm-dis.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-nm: t:\1\bin\llvm-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-readtapi: t:\1\bin\llvm-readtapi.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-size: t:\1\bin\llvm-size.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sourcekitd-test: t:\1\bin\sourcekitd-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using complete-test: t:\1\bin\complete-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-api-digester: t:\1\bin\swift-api-digester.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-cache-tool: t:\1\bin\swift-cache-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-refactor: t:\1\bin\swift-refactor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-demangle-yamldump: t:\1\bin\swift-demangle-yamldump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-demangle: t:\1\bin\swift-demangle.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_O' program, try setting BENCHMARK_O in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_Driver' program, try setting BENCHMARK_DRIVER in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using wasm-ld: t:\1\bin\wasm-ld.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-plugin-server: t:\1\bin\swift-plugin-server.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-parse-test: t:\1\bin\swift-parse-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:385: note: Using resource dir: ..\lib\swift
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:432: note: Compiling with -swift-version 4
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:528: note: Using Clang module cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\clang-module-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:529: note: Using test results dir: T:/1/./swift-test-results/x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:534: note: Using code completion cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\completion-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:1496: note: Testing Windows x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:2253: note: Using platform module dir: ..\lib\swift\windows
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-reflection-test-windows-x86_64: t:\1\bin\swift-reflection-test-windows-x86_64.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:2861: note: Available features: CMAKE_GENERATOR=Ninja, CODEGENERATOR=AArch64, CODEGENERATOR=ARM, CODEGENERATOR=WebAssembly, CODEGENERATOR=X86, CPU=x86_64, MSVC_VER=17.0, OS=windows-msvc, PTRSIZE=64, SWIFT_VERSION=4, VENDOR=unknown, asserts, concurrency, concurrency_runtime, cplusplus_driver, differentiable_programming, distributed, executable_test, no_asan, no_lto, nonexecutable_test, objc_codegen, observation, optimized_stdlib, optimized_stdlib_x86_64, reflection, sourcekit, string_processing, swift_only_stable_abi, swift_stdlib_no_asserts, swift_swift_parser, swift_test_mode_optimize_none, swift_test_mode_optimize_none_x86_64, swift_tools_extra, thread_safe_runtime, threading_win32

Good Run:

lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:210: note: Using cmake: C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/bin/cmake.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-frontend: t:\1\bin\swift-frontend.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift: t:\1\bin\swift.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swiftc: t:\1\bin\swiftc.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-opt: t:\1\bin\sil-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-func-extractor: t:\1\bin\sil-func-extractor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-llvm-gen: t:\1\bin\sil-llvm-gen.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-nm: t:\1\bin\sil-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-passpipeline-dumper: t:\1\bin\sil-passpipeline-dumper.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using lldb-moduleimport-test: t:\1\bin\lldb-moduleimport-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-ide-test: t:\1\bin\swift-ide-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-dependency-tool: t:\1\bin\swift-dependency-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-reflection-dump: t:\1\bin\swift-reflection-dump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-remoteast-test: t:\1\bin\swift-remoteast-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-indent: t:\1\bin\swift-indent.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-symbolgraph-extract: t:\1\bin\swift-symbolgraph-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-api-extract: t:\1\bin\swift-api-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using clang: t:\1\bin\clang.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using clang++: t:\1\bin\clang++.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-link: t:\1\bin\llvm-link.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-llvm-opt: t:\1\bin\swift-llvm-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-profdata: t:\1\bin\llvm-profdata.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-cov: t:\1\bin\llvm-cov.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-strings: t:\1\bin\llvm-strings.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using FileCheck: t:\1\bin\filecheck.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-dwarfdump: t:\1\bin\llvm-dwarfdump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-readelf: t:\1\bin\llvm-readelf.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-dis: t:\1\bin\llvm-dis.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-nm: t:\1\bin\llvm-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-readtapi: t:\1\bin\llvm-readtapi.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-size: t:\1\bin\llvm-size.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sourcekitd-test: t:\1\bin\sourcekitd-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using complete-test: t:\1\bin\complete-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-api-digester: t:\1\bin\swift-api-digester.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-cache-tool: t:\1\bin\swift-cache-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-refactor: t:\1\bin\swift-refactor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-demangle-yamldump: t:\1\bin\swift-demangle-yamldump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-demangle: t:\1\bin\swift-demangle.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_O' program, try setting BENCHMARK_O in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_Driver' program, try setting BENCHMARK_DRIVER in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using wasm-ld: t:\1\bin\wasm-ld.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-plugin-server: t:\1\bin\swift-plugin-server.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-parse-test: t:\1\bin\swift-parse-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:385: note: Using resource dir: t:\1\lib\swift
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:432: note: Compiling with -swift-version 4
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:528: note: Using Clang module cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\clang-module-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:529: note: Using test results dir: T:/1/./swift-test-results/x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:534: note: Using code completion cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\completion-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:1496: note: Testing Windows x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:2253: note: Using platform module dir: t:\1\lib\swift\windows
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-reflection-test-windows-x86_64: t:\1\bin\swift-reflection-test-windows-x86_64.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:2861: note: Available features: CMAKE_GENERATOR=Ninja, CODEGENERATOR=AArch64, CODEGENERATOR=ARM, CODEGENERATOR=WebAssembly, CODEGENERATOR=X86, CPU=x86_64, MSVC_VER=17.0, OS=windows-msvc, PTRSIZE=64, SWIFT_VERSION=4, VENDOR=unknown, asserts, concurrency, concurrency_runtime, cplusplus_driver, differentiable_programming, distributed, executable_test, no_asan, no_lto, nonexecutable_test, objc_codegen, observation, optimized_stdlib, optimized_stdlib_x86_64, reflection, sourcekit, string_processing, swift_only_stable_abi, swift_stdlib_no_asserts, swift_swift_parser, swift_test_mode_optimize_none, swift_test_mode_optimize_none_x86_64, swift_tools_extra, thread_safe_runtime, threading_win32

This seems to be the diff:

- lit.py: swift\test\lit.cfg:289: note: using swift: t:\1\bin\swift.exe
- lit.py: swift\test\lit.cfg:385: note: Using resource dir: t:\1\lib\swift
+ lit.py: swift\test\lit.cfg:385: note: Using resource dir: ..\lib\swift
- lit.py: swift\test\lit.cfg:2253: note: Using platform module dir: t:\1\lib\swift\windows
+ lit.py: swift\test\lit.cfg:2253: note: Using platform module dir: ..\lib\swift\windows

This does match the majority of the failures. Now the question is, why is the resource dir and platform module dir path different?