Sync with Khronos SPIR-V Translator 4805372e24

Last patch is called:
  "Fix invalid .spv->.bc translation of Intel FPGA merge attribute (#291)"

Conflicted with:
  "[SYCL] Enable build and LIT tests on windows (#463)"
Resolved to "Ours".

Commits excluded from the sync as it breakes SYCL build:
  "Use std::make_unique after LLVM change r369130"
  "Move to C++14"
both of these changes should be cherry-pick after our next LLVM update.

Signed-off-by: Dmitry Sidorov <[email protected]>

Author: MrSidims

llvm-spirv/CMakeLists.txt

@@ -63,6 +63,8 @@ endif(LLVM_INCLUDE_TESTS)
 install(
   FILES
     ${LLVM_SPIRV_INCLUDE_DIRS}/LLVMSPIRVLib.h
+    ${LLVM_SPIRV_INCLUDE_DIRS}/LLVMSPIRVOpts.h
+    ${LLVM_SPIRV_INCLUDE_DIRS}/LLVMSPIRVExtensions.inc
   DESTINATION
     ${CMAKE_INSTALL_PREFIX}/include/LLVMSPIRVLib
 )

llvm-spirv/README.md

@@ -18,7 +18,7 @@ The files/directories related to the translator:
 
 ## Build Instructions
 
-Master branch of this repo is aimed to be buildable with the latest LLVM version.
+The `master` branch of this repo is aimed to be buildable with the latest LLVM `master` or `trunk` revision.
 
 ### Build with pre-installed LLVM
 
@@ -64,7 +64,8 @@ make llvm-spirv -j`nproc`
 
 ## Test instructions
 
-All tests related to the translator are placed in the [test](test) directory. Optionally the tests can make use of spirv-val (part of SPIRV-Tools) in order to validate the generated SPIR-V against the official SPIR-V specification.
+All tests related to the translator are placed in the [test](test) directory. A number of the tests require spirv-as (part of SPIR-V Tools) to run, but the remainder of the tests can still be run without this. Optionally the tests can make use of spirv-val (part of SPIRV-Tools) in order to validate the generated SPIR-V against the official SPIR-V specification.
+
 In case tests are failing due to SPIRV-Tools not supporting certain SPIR-V features, please get an updated package. The `PKG_CONFIG_PATH` environmental variable can be used to let cmake point to a custom installation.
 
 Execute the following command inside the build directory to run translator tests:
@@ -99,10 +100,52 @@ To translate between LLVM IR and SPIR-V:
     * `-spirv-text` - read/write SPIR-V in an internal textual format for debugging purpose. The textual format is not defined by SPIR-V spec.
     * `-help` - to see full list of options
 
+### Handling SPIR-V versions generated by the translator
+
+There is one option to control the behavior of the translator with respect to
+the version of the SPIR-V file which is being generated/consumed.
+
+* `-spirv-max-version=` - this option allows restricting the
+  SPIRV-LLVM-Translator **not** to generate a SPIR-V with a version which is
+  higher than the one specified via this option.
+
+  If the `-r` option was also specified, the SPIRV-LLVM-Translator will reject
+  the input file and emit an error if the SPIR-V version in it is higher than
+  one specified via this option.
+
+Allowed values are `1.0`/`1.1`.
+
+More information can be found in
+[SPIR-V versions and extensions handling](docs/SPIRVVersionsAndExtensionsHandling.rst)
+
+### Handling SPIR-V extensions generated by the translator
+
+By default, during SPIR-V generation, the translator doesn't use any extensions.
+However, during SPIR-V consumption, the translator accepts input files that use
+any known extensions.
+
+If certain extensions are required to be enabled or disabled, the following
+command line option can be used:
+
+* ``--spirv-ext=`` - this options allows controlling which extensions are
+  allowed/disallowed
+
+Valid value for this option is comma-separated list of extension names prefixed
+with ``+`` or ``-`` - plus means allow to use extension, minus means disallow
+to use extension. There is one more special value which can be used as extension
+name in this option: ``all`` - it affects all extension which are known to the
+translator.
+
+If ``--spirv-ext`` contains the name of an extension which is not known for the
+translator, it will emit an error.
+
+More information can be found in
+[SPIR-V versions and extensions handling](docs/SPIRVVersionsAndExtensionsHandling.rst)
+
 ## Branching strategy
 
 Code on the master branch in this repository is intended to be compatible with master/trunk branch of the [llvm](https://github.com/llvm-mirror/llvm) project. That is, for an OpenCL kernel compiled to llvm bitcode by the latest version(built with the latest git commit or svn revision) of Clang it should be possible to translate it to SPIR-V with the llvm-spirv tool.
 
 All new development should be done on the master branch.
 
-To have versions compatible with released versions of LLVM and Clang, corresponding branches are created in this repository. For example, to build translator with LLVM 7.0 ([release_70](https://github.com/llvm-mirror/llvm/tree/release_70)) one should use [llvm_release_70](https://github.com/KhronosGroup/SPIRV-LLVM-Translator/tree/llvm_release_70) branch.
+To have versions compatible with released versions of LLVM and Clang, corresponding branches are available in this repository. For example, to build the translator with LLVM 7.0 ([release_70](https://github.com/llvm-mirror/llvm/tree/release_70)) one should use the [llvm_release_70](https://github.com/KhronosGroup/SPIRV-LLVM-Translator/tree/llvm_release_70) branch. As a general rule, commits from the master branch may be backported to the release branches as long as they do not depend on features from a later LLVM/Clang release and there are no objections from the maintainer(s). There is no guarantee that older release branches are proactively kept up to date with master, but you can request certain commits on older release branches by creating a pull request or raising an issue on GitHub.

llvm-spirv/docs/SPIRVVersionsAndExtensionsHandling.rst

@@ -0,0 +1,243 @@
+=======================================
+SPIR-V versions and extensions handling
+=======================================
+
+.. contents::
+   :local:
+
+Overview
+========
+
+This document describes how the translator makes decisions about using
+instructions from different version of the SPIR-V core and extension
+specifications.
+
+Being able to control the resulting SPIR-V version is important: the target
+consumer might be quite old, without support for new SPIR-V versions and there
+must be the possibility to control which version of the SPIR-V specification
+that will be used during translation.
+
+SPIR-V extensions is another thing which must be controllable. Extensions
+can update and re-define semantics and validation rules for existing SPIR-V
+entries and it is important to ensure that the translator is able to generate
+valid SPIR-V according to the core spec, without uses of any extensions if such
+SPIR-V was requested by user.
+
+For example, without such infrastructure it is impossible to disable use of
+``SPV_KHR_no_integer_wrap_decoration`` - it will be always generated if
+corresponding LLVM IR counterparts are encountered in input module.
+
+It is worth mentioning that SPIR-V versions and extensions the handling of
+SPIR-V versions and extension is mostly important for the SPIR-V generation
+step. On the consumer side it is the responsibility of the consumer to analyze
+the incoming SPIR-V file and reject it if it contains something that is not
+supported by the consumer.
+
+However, translator simplifies this step for downstream users by checking
+version and extensions in SPIR-V module during ``readSpirv``/``readSpirvModule``
+phases.
+
+SPIR-V Versions
+===============
+
+SPIR-V Generation step
+----------------------
+
+By default translator selects version of generated SPIR-V file based on features
+used in this file. For example, if it contains ``dereferencable`` LLVM IR
+attribute, ``MaxByteOffset`` decoration will be generated and resulting SPIR-V
+version will be raised to 1.1.
+
+.. note::
+   There is no documentation about which exact features from newest
+   SPIR-V spec versions will be used by the translator. If you are interested
+   when or why a particular SPIR-V instruction is generated, please check this
+   in the source code. Consider this as an implementation detail and if you
+   disagree with something, you can always open an issue or submit pull request
+   - contributions are welcome!
+
+There is one option to control the behavior of the translator with respect to
+the version of the SPIR-V file which is being generated/consumed.
+
+* ``--spirv-max-version=`` - instructs the translator to generate SPIR-V file
+  corresponding to any spec version which is less than or equal to the
+  specified one. Behavior of the translator is the same as by default with only
+  one exception: resulting SPIR-V version cannot be raised higher than
+  specified by this option.
+
+Allowed values are ``1.0`` and ``1.1``.
+
+.. warning::
+   These two options are mutually exclusive and cannot be specified at the
+   same time.
+
+If the translator encounters something that cannot be represented by set of
+allowed SPIR-V versions (which might contain only one version), it does one of
+the following things:
+
+* ignores LLVM IR entity in the input file.
+
+  For example, ``dereferencable`` LLVM IR attribute can be ignored if it is not
+  allowed to generate SPIR-V 1.1 and higher.
+
+* tries to represent LLVM IR entity with allowed instructions.
+
+  For example, ``OpPtrEqual`` can be used if SPIR-V 1.4 is not allowed and can
+  be emulated via ``OpConvertPtrToU`` + ``OpIEqual`` sequence.
+
+* emits error if LLVM IR entity cannot be ignored and cannot be emulated using
+  available instructions.
+
+  For example, if global constructors/destructors
+  (represented by @llvm.global_ctors/@llvm.global_dtors) are present in a module
+  then the translator should emit error if it cannot use SPIR-V 1.1 and higher
+  where ``Initializer`` and ``Finalizer`` execution modes are described.
+
+SPIR-V Consumption step
+-----------------------
+
+By default, translator consumes SPIR-V of any version which is supported.
+
+This behavior, however, can be controlled via the same switches described in
+the previous section.
+
+If one of the switches present and translator encountered SPIR-V file
+corresponding to a spec version which is not included into set of allowed
+SPIR-V versions, translator emits error.
+
+SPIR-V Extensions
+=================
+
+SPIR-V Generation step
+----------------------
+
+By default, translator doesn't use any extensions. If it required to enable
+certain extension, the following command line option can be used:
+
+* ``--spirv-ext=`` - allows to control list of allowed/disallowed extensions.
+
+Valid value for this option is comma-separated list of extension names prefixed
+with ``+`` or ``-`` - plus means allow to use extension, minus means disallow
+to use extension. There is one more special value which can be used as extension
+name in this option: ``all`` - it affects all extension which are known to the
+translator.
+
+If ``--spirv-ext`` contains name of extension which is not know for the
+translator, it will emit error.
+
+Examples:
+
+* ``--spirv-ext=+SPV_KHR_no_integer_wrap_decoration,+SPV_INTEL_subgroups``
+* ``--spirv-ext=+all,-SPV_INTEL_fpga_loop_controls``
+
+.. warning::
+   Extension name cannot be allowed and disallowed at the same time: for inputs
+   like ``--spirv-ext=+SPV_INTEL_subgroups,-SPV_INTEL_subgroups`` translator
+   will emit error about invalid arguments.
+
+.. note::
+   Since by default during SPIR-V generation all extensions are disabled, this
+   means that ``-all,`` is implicitly added at the beggining of the
+   ``-spirv-ext`` value.
+
+If the translator encounters something that cannot be represented by set of
+allowed SPIR-V extensions (which might be empty), it does one of the following
+things:
+
+* ignores LLVM IR entity in the input file.
+
+  For example, ``nsw``/``nuw`` LLVM IR attributes can be ignored if it is not
+  allowed to generate SPIR-V 1.4 and ``SPV_KHR_no_integer_wrap_decoration``
+  extension is disallowed.
+
+* tries to represent LLVM IR entity with allowed instructions.
+
+  Translator could translate calls to a new built-in functions defined by some
+  extensions as usual call instructions without using special SPIR-V
+  instructions.
+
+  However, this could result in a strange SPIR-V and most likely will lead to
+  errors during consumption. Having that, translator should emit errors if it
+  encounters a call to a built-in function from an extension which must be
+  represented as a special SPIR-V instruction from extension which wasn't
+  allowed to be used. I.e. if translator knows that this certain LLVM IR entity
+  belongs to an extension functionality and this extension is disallowed, it
+  should emit error rather than emulating it.
+
+* emits error if LLVM IR entity cannot be ignored and cannot be emulated using
+  available instructions.
+
+  For example, new built-in types defined by
+  ``cl_intel_device_side_avc_motion_estimation`` cannot be represented in SPIR-V
+  if ``SPV_INTEL_device_side_avc_motion_estimation`` is disallowed.
+
+SPIR-V Consumption step
+-----------------------
+
+By default, translator consumes SPIR-V regardless of list extensions which are
+used by the input file, i.e. all extensions are allowed by default during
+consumption step.
+
+.. note::
+   This is opposite to the generation step and this is done on purpose: to not
+   broke workflows of existing users of the translator.
+
+.. note::
+   Since by default during SPIR-V consumption all extensions are enabled, this
+   means that ``+all,`` is implicitly added at the beggining of the
+   ``-spirv-ext`` value.
+
+This behavior, however, can be controlled via the same switches described in
+the previous section.
+
+If ``--spirv-ext`` switch presents, translator will emit error if it finds out
+that input SPIR-V file uses disallowed extension.
+
+.. note::
+   If the translator encounters unknown extension in the input SPIR-V file, it
+   will emit error regardless of ``-spirv-ext`` option value.
+
+If one of the switches present and translator encountered SPIR-V file
+corresponding to a spec version which is not included into set of allowed
+SPIR-V versions, translator emits error.
+
+How to control translator behavior when using it as library
+===========================================================
+
+When using translator as library it can be controlled via bunch of alternative
+APIs that have additional argument: ``TranslatorOpts`` object which
+encapsulates information about available SPIR-V versions and extensions.
+
+List of new APIs is: ``readSpirvModule``, ``writeSpirv`` and ``readSpirv``.
+
+.. note::
+   See ``LLVMSPIRVOpts.h`` for more details.
+
+How to get ``TranslatorOpts`` object
+------------------------------------
+
+1. Default constructor. Equal to:
+
+   ``--spirv-max-version=MaxKnownVersion --spirv-ext=-all``
+
+   .. note::
+      There is method ``TranslatorOpts::enableAllExtensions()`` that allows you
+      to quickly enable all known extensions if it is needed.
+
+2. Constructor which accepts all parameters
+
+   Consumes both max SPIR-V version and optional map with extensions status
+   (i.e. which one is allowed and which one is disallowed)
+
+Extensions status map
+^^^^^^^^^^^^^^^^^^^^^
+
+This map is defined as ``std::map<ExtensionID, bool>`` and it is intended to
+show which extension is allowed to be used (``true`` as value) and which is not
+(``false`` as value).
+
+.. note::
+   If certain ``ExtensionID`` value is missed in the map, it automatically means
+   that extension is not allowed to be used.
+
+   This implies that by default, all extensions are disallowed.

llvm-spirv/include/LLVMSPIRVExtensions.inc

@@ -0,0 +1,15 @@
+
+#ifndef EXT
+  #error "EXT macro must be defined"
+#endif
+
+EXT(SPV_KHR_no_integer_wrap_decoration)
+EXT(SPV_INTEL_subgroups)
+EXT(SPV_INTEL_media_block_io)
+EXT(SPV_INTEL_device_side_avc_motion_estimation)
+EXT(SPV_INTEL_fpga_loop_controls)
+EXT(SPV_INTEL_fpga_memory_attributes)
+EXT(SPV_INTEL_unstructured_loop_controls)
+EXT(SPV_INTEL_fpga_reg)
+EXT(SPV_INTEL_blocking_pipes)
+EXT(SPV_INTEL_function_pointers)

llvm-spirv/include/LLVMSPIRVLib.h

@@ -41,6 +41,8 @@
 #ifndef SPIRV_H
 #define SPIRV_H
 
+#include "LLVMSPIRVOpts.h"
+
 #include <iostream>
 #include <string>
 
@@ -66,6 +68,7 @@ void initializePreprocessMetadataPass(PassRegistry &);
 #include "llvm/IR/Module.h"
 
 namespace SPIRV {
+
 class SPIRVModule;
 
 /// \brief Check if a string contains SPIR-V binary.
@@ -94,6 +97,12 @@ bool isSpirvText(std::string &Img);
 std::unique_ptr<SPIRVModule> readSpirvModule(std::istream &IS,
                                              std::string &ErrMsg);
 
+/// \brief Load SPIR-V from istream as a SPIRVModule.
+/// \returns null on failure.
+std::unique_ptr<SPIRVModule> readSpirvModule(std::istream &IS,
+                                             const SPIRV::TranslatorOpts &Opts,
+                                             std::string &ErrMsg);
+
 } // End namespace SPIRV
 
 namespace llvm {
@@ -107,6 +116,16 @@ bool writeSpirv(Module *M, std::ostream &OS, std::string &ErrMsg);
 bool readSpirv(LLVMContext &C, std::istream &IS, Module *&M,
                std::string &ErrMsg);
 
+/// \brief Translate LLVM module to SPIR-V and write to ostream.
+/// \returns true if succeeds.
+bool writeSpirv(Module *M, const SPIRV::TranslatorOpts &Opts, std::ostream &OS,
+                std::string &ErrMsg);
+
+/// \brief Load SPIR-V from istream and translate to LLVM module.
+/// \returns true if succeeds.
+bool readSpirv(LLVMContext &C, const SPIRV::TranslatorOpts &Opts,
+               std::istream &IS, Module *&M, std::string &ErrMsg);
+
 /// \brief Convert a SPIRVModule into LLVM IR.
 /// \returns null on failure.
 std::unique_ptr<Module>