Failed to load image extension

[vedantroy]

🐛 Describe the bug

import torchvision

gives the warning:

/home/ray/anaconda3/lib/python3.9/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: /home/ray/anaconda3/lib/python3.9/site-packages/torchvision/image.so: undefined symbol: _ZN3c106detail19maybe_wrap_dim_slowEllb

Versions

Collecting environment information...
PyTorch version: 1.14.0.dev20221027+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-53-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce GTX 1080 Ti
GPU 1: NVIDIA GeForce GTX 1080 Ti

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-boto3-cloudformation==1.25.4
[pip3] mypy-boto3-dynamodb==1.25.0
[pip3] mypy-boto3-ec2==1.25.5
[pip3] mypy-boto3-lambda==1.25.0
[pip3] mypy-boto3-rds==1.25.1
[pip3] mypy-boto3-s3==1.25.0
[pip3] mypy-boto3-sqs==1.25.0
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.5
[pip3] torch==1.14.0.dev20221027+cu116
[pip3] torch-tb-profiler==0.4.0
[pip3] torchdata==0.6.0.dev20221027
[pip3] torchsnapshot-nightly==2022.10.29
[pip3] torchvision==0.15.0a0+0dceac0
[conda] numpy 1.23.5 pypi_0 pypi
[conda] torch 1.14.0.dev20221027+cu116 pypi_0 pypi
[conda] torch-tb-profiler 0.4.0 pypi_0 pypi
[conda] torchdata 0.6.0.dev20221027 pypi_0 pypi
[conda] torchsnapshot-nightly 2022.10.29 pypi_0 pypi
[conda] torchvision 0.15.0a0+0dceac0 pypi_0 pypi

[vedantroy]

Is there a guideline for what the matching version of torchvision should be for a given torch commit?

[pmeier]

In general, the master of PyTorch is compatible with the main of TorchVision. Nightlies are compatible for the same day. You have

[pip3] torch==1.14.0.dev20221027+cu116

and

[conda] torchvision 0.15.0a0+0dceac0 pypi_0 pypi

I'm guessing this a bug in the collection and you actually installed torchvision from source, right? Commit should be 0dceac0, which was only merged a couple of hours ago. Meaning, the compatible PyTorch nightly version is torch==1.14.0.dev20221213.

Could you update that and see if the error persists?

[vedantroy]

@pmeier Thanks, will try it out. How did you figure out the commit for torch from this information by the way?

[conda] torch 1.14.0.dev20221027+cu116 pypi_0 pypi

does not contain the commit information for pytorch.
And, how did you go from the pytorch commit to the compatible torchvision version?

And yes, I installed torchvision from source.

[pmeier]

How did you figure out the commit for torch from this information by the way?

I didn't. 0dceac0 is a torchvision commit. The information you get here is the date of the nightly, i.e. 20221027 -> Oct 27, 2022. With that you can go the nightly branch of pytorch/pytorch and look up the commit. For the example above, this is pytorch/pytorch@21bef8e.

And, how did you go from the pytorch commit to the compatible torchvision version?

Installing from source gives you the first part of the commit hash that was build in the version, i.e. 0dceac0. If you append that to https://github.com/pytorch/vision/commit/, e.g. 0dceac0, you can find it on GH. Looking at the nightly branch for torchvision, you find it for Dec 13, 2022. Meaning, the compatible torch nightly is the one from the same day, i.e. torch==1.14.0.dev20221213.

Please note, that this lookup from commit to nightly date is not guaranteed to work. Above we got lucky since the commit we were looking for was actually the last one that was included in that nightly. In general that does not need to be the case. So you often need some back and forth to find the correct date.

[vedantroy]

Hm, there's no torch with version 1.14.0.dev20221213+cu116, I can probably build without cuda support -- but -- I do need cuda.

Not sure what to do here. Will try to build with cuda+117 and see if that helps.

Update: no cuda+117 either.

[pmeier]

My bad, they switched the versioning scheme for the upcoming 2.0 release. The nightly you are looking for is torch-2.0.0.dev20221213+cu116. You can

pip install https://download.pytorch.org/whl/nightly/cu116/torch-2.0.0.dev20221213%2Bcu116-cp39-cp39-linux_x86_64.whl

[vedantroy]

Still getting the error. It also seems like packages like torch==1.14.0.dev20221027+cu116 have been removed from pypi?

New collect_env output:

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-boto3-cloudformation==1.25.4
[pip3] mypy-boto3-dynamodb==1.25.0
[pip3] mypy-boto3-ec2==1.25.5
[pip3] mypy-boto3-lambda==1.25.0
[pip3] mypy-boto3-rds==1.25.1
[pip3] mypy-boto3-s3==1.25.0
[pip3] mypy-boto3-sqs==1.25.0
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.5
[pip3] pytorch-triton==2.0.0+0d7e753227
[pip3] torch==2.0.0.dev20221213+cu116
[pip3] torch-tb-profiler==0.4.0
[pip3] torchdata==0.6.0.dev20230101
[pip3] torchsnapshot-nightly==2022.10.29
[pip3] torchtriton==2.0.0+0d7e753227
[pip3] torchvision==0.15.0a0+edb3a80
[conda] numpy                     1.23.5                   pypi_0    pypi
[conda] pytorch-triton            2.0.0+0d7e753227          pypi_0    pypi
[conda] torch                     2.0.0.dev20221213+cu116          pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchdata                 0.6.0.dev20230101          pypi_0    pypi
[conda] torchsnapshot-nightly     2022.10.29               pypi_0    pypi
[conda] torchtriton               2.0.0+0d7e753227          pypi_0    pypi
[conda] torchvision               0.15.0a0+edb3a80          pypi_0    pypi

[pmeier]

Still getting the error.

Your env is still broken. The torchvision commit is from Oct 25, 2022 while the torch nightly is from Dec 13, 2022. Please get that sorted out for example by a clean install before we debug any further.

It also seems like packages like torch==1.14.0.dev20221027+cu116 have been removed from pypi?

The nightly releases have never been on PyPI, but only on our index. This is why you have to use the --extra-index-url option when installing.

[vedantroy]

Here's a fixed version (I think):

pip3] torch==2.0.0.dev20230120+cu116
[pip3] torch-tb-profiler==0.4.0
[pip3] torchdata==0.6.0.dev20230101
[pip3] torchsnapshot-nightly==2022.10.29
[pip3] torchvision==0.15.0a0+d2d448c
[conda] numpy                     1.23.5                   pypi_0    pypi
[conda] pytorch-triton            2.0.0+0d7e753227          pypi_0    pypi
[conda] torch                     2.0.0.dev20230120+cu116          pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchdata                 0.6.0.dev20230101          pypi_0    pypi
[conda] torchsnapshot-nightly     2022.10.29               pypi_0    pypi
[conda] torchvision               0.15.0a0+d2d448c          pypi_0    pypi
Pillow/Pillow-SIMD version: 9.0.0.post1

torchvision is using commit: d2d448c, which I believe is a commit on January 20th at 7:58 AM EST.
Meanwhile, torch is using 20230120, which I think is "2023-01-20".

Yet, I'm still getting the same error.

@pmeier

[pmeier]

Thanks for confirming. Could you look for the image.so file in the installed torchvision folder? It should be directly in there like lib/python3.X/site-packages/torchvision/image.so. Or if you used an editable install, torchvision/image.so in the repository. If it is there, could you post the output of ldd image.so?

[vedantroy]

Thanks for confirming. Could you look for the image.so file in the installed torchvision folder? It should be directly in there like lib/python3.X/site-packages/torchvision/image.so. Or if you used an editable install, torchvision/image.so in the repository. If it is there, could you post the output of ldd image.so?

Here it is:

        linux-vdso.so.1 (0x00007fff693ed000)
        libpng16.so.16 => /home/ray/anaconda3/lib/libpng16.so.16 (0x00007f7a0e3f1000)
        libjpeg.so.8 => /home/ray/anaconda3/lib/libjpeg.so.8 (0x00007f7a0e345000)
        libc10.so => not found
        libtorch_cpu.so => not found
        libstdc++.so.6 => /home/ray/anaconda3/lib/libstdc++.so.6 (0x00007f7a0e12b000)
        libgcc_s.so.1 => /home/ray/anaconda3/lib/libgcc_s.so.1 (0x00007f7a0e112000)
        libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7a0e0ed000)
        libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007f7a0defb000)
        libz.so.1 => /home/ray/anaconda3/lib/./libz.so.1 (0x00007f7a0dedd000)
        libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007f7a0dd8e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7a0e44d000)