ggml : add DirectML backend

[ggerganov]

It seems like DirectML supports the upcoming NPU-enabled chips for Windows machines:
https://devblogs.microsoft.com/directx/introducing-neural-processor-unit-npu-support-in-directml-developer-preview/

I don't think there is any other way to tap into this hardware, so we should explore if it possible to add this library as a backend in ggml in order to run stuff on the NPUs. There has been some semi-related work in the past that combined ggml and Direct3D: https://github.com/Const-me/Whisper. Not sure if it is relevant at all, maybe just as an inspiration

[arch-btw]

Great idea, it looks like a lot of the upcoming AI hardware is going to have NPUs.

[slaren]

I am not convinced that a DirectML backend is possible, the operators are too high level and new ones cannot be added. This means that we cannot implement a matrix multiplication operator that supports our quant formats. It might be possible to do it with DirectX 12 shaders, but at that point it would be a DirectX 12 backend more than a DirectML backend. It would not allow using onnx models regardless.

[ggerganov]

Would using DirectX 12 shaders allow us to run stuff on the NPU? I suppose no, but just making sure. The main point of a potential DirectML backend would be to utilize the NPU. If it is too high-level (i.e. something like what CoreML is on Apple Silicon) then I agree it's not worth or possible to add support for it

[slaren]

I am not sure it is possible to create custom NPU kernels at all. https://github.com/openvinotoolkit/npu_plugin seems to contain a compiler for the Intel NPU, but it's not clear if it is complete, and they have removed the source of the kernels that should be located in https://github.com/openvinotoolkit/npu_plugin/tree/develop/sw_runtime_kernels, leaving only the binary blobs.

[sinni800]

Interesting, but they have a PyTorch implementation, I thought PyTorch is pretty diverse in what it can support, but I have not much insight into the base here. Or is PyTorch automatically supplementing what isn't supported by DirectML with CPU?

[kylo5aby]

Interesting, but they have a PyTorch implementation, I thought PyTorch is pretty diverse in what it can support, but I have not much insight into the base here. Or is PyTorch automatically supplementing what isn't supported by DirectML with CPU?

it should automatically fallback to CPU in this case.

[woachk]

@slaren the lower level D3D metacommands interface leveraged by DirectML is not publicly documented.

The Intel NPU d3d12 drivers have a shader compiler and accept custom kernels. But the DirectML driver for the NPU on Qualcomm systems is metacommands only, w/ no custom kernel support, at least so far.

[slaren]

@woachk thanks, that's useful information. If the Intel NPU driver accepts custom kernels via d3d12 shaders, I expect that it would be possible to fully support it through a d3d12 backend. For NPUs that only support DirectML, it may still be possible to support fp16 and fp32 models. It may also be possible to create a backend that transparently quantizes the tensors to an internal format, and in this way it may be possible to support the quantization type of DirectML, although that would be a significant deviation from the current backends. Personally I don't think that a backend that cannot use the ggml quantization types would be very useful.