Create .so symlinks for driver libraries in container #326

elezar · 2024-02-01T09:26:38Z

This change explicitly creates .so and SONAME symlinks for injected driver libraries. This means that the create-soname-symlinks hook that was added in #947 is not longer required and is therefore removed.

$ docker run --rm -ti --runtime=nvidia --gpus all busybox  ls -al /usr/lib/x86_64-linux-gnu/
total 263564
drwxr-xr-x    2 root     root          4096 Jul 21 10:00 .
drwxr-xr-x    3 root     root          4096 Jul 21 10:00 ..
lrwxrwxrwx    1 root     root            12 Jul 21 10:00 libcuda.so -> libcuda.so.1
lrwxrwxrwx    1 root     root            21 Jul 21 10:00 libcuda.so.1 -> libcuda.so.570.133.20
-rw-r--r--    1 root     root      71365624 Apr 13 04:46 libcuda.so.570.133.20
lrwxrwxrwx    1 root     root            20 Jul 21 10:00 libcudadebugger.so -> libcudadebugger.so.1
lrwxrwxrwx    1 root     root            29 Jul 21 10:00 libcudadebugger.so.1 -> libcudadebugger.so.570.133.20
-rw-r--r--    1 root     root      10244640 Apr 13 04:17 libcudadebugger.so.570.133.20
lrwxrwxrwx    1 root     root            17 Jul 21 10:00 libnvidia-ml.so -> libnvidia-ml.so.1
lrwxrwxrwx    1 root     root            26 Jul 21 10:00 libnvidia-ml.so.1 -> libnvidia-ml.so.570.133.20
-rw-r--r--    1 root     root       2217912 Apr 13 04:23 libnvidia-ml.so.570.133.20
lrwxrwxrwx    1 root     root            19 Jul 21 10:00 libnvidia-nvvm.so -> libnvidia-nvvm.so.4
lrwxrwxrwx    1 root     root            28 Jul 21 10:00 libnvidia-nvvm.so.4 -> libnvidia-nvvm.so.570.133.20
-rw-r--r--    1 root     root      81978912 Apr 13 05:03 libnvidia-nvvm.so.570.133.20
lrwxrwxrwx    1 root     root            21 Jul 21 10:00 libnvidia-opencl.so -> libnvidia-opencl.so.1
lrwxrwxrwx    1 root     root            30 Jul 21 10:00 libnvidia-opencl.so.1 -> libnvidia-opencl.so.570.133.20
-rw-r--r--    1 root     root      65758768 Apr 13 04:46 libnvidia-opencl.so.570.133.20
-rw-r--r--    1 root     root         10176 Apr 13 04:21 libnvidia-pkcs11-openssl3.so.570.133.20
-rw-r--r--    1 root     root         10168 Apr 13 04:21 libnvidia-pkcs11.so.570.133.20
lrwxrwxrwx    1 root     root            29 Jul 21 10:00 libnvidia-ptxjitcompiler.so -> libnvidia-ptxjitcompiler.so.1
lrwxrwxrwx    1 root     root            38 Jul 21 10:00 libnvidia-ptxjitcompiler.so.1 -> libnvidia-ptxjitcompiler.so.570.133.20
-rw-r--r--    1 root     root      38251952 Apr 13 04:33 libnvidia-ptxjitcompiler.so.570.133.20

And:

$ docker run --rm -ti --runtime=nvidia --gpus all ubuntu ldconfig -p | grep libcuda
        libcudadebugger.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcudadebugger.so.1
        libcudadebugger.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcudadebugger.so
        libcuda.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcuda.so.1
        libcuda.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcuda.so

showing that the .so and .so.1 entries are present in the ldcache.

klueska

Initial pass. I lost some steam near the end, but wanted to drop the comments I had for now.

cmd/nvidia-ctk/hook/create-dot-so-symlinks/create-dot-so-symlinks.go

internal/config/features.go

internal/lookup/root/root.go

elezar

I think some of the issues were because this was out of date.

cmd/nvidia-ctk/hook/create-dot-so-symlinks/create-dot-so-symlinks.go

internal/config/features.go

ArangoGutierrez · 2025-02-25T17:56:40Z

@elezar does #906 supersede this one?

elezar · 2025-02-25T20:28:04Z

@elezar does #906 supersede this one?

No. This is not related to the CUDA Forward Compat libraries at all. This is about creating .so -> SONAME symlinks for driver libraries in the container. #906 is about ensuring that the CUDA Forward compat libraries are properly detectable in a container if required.

coveralls · 2025-07-17T09:54:17Z

Pull Request Test Coverage Report for Build 16469131076

Details

70 of 99 (70.71%) changed or added relevant lines in 4 files are covered.
16 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.4%) to 35.391%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/nvcdi/driver-nvml.go	0	1	0.0%
internal/discover/symlinks.go	68	96	70.83%

Files with Coverage Reduction	New Missed Lines	%
internal/ldconfig/ldconfig.go	2	0.0%
internal/discover/list.go	14	0.0%

Totals
Change from base Build 16421142187:	0.4%
Covered Lines:	4511
Relevant Lines:	12746

💛 - Coveralls

cmd/nvidia-ctk/info/info.go

pkg/nvcdi/driver-nvml.go

internal/discover/symlinks.go

klueska · 2025-07-22T20:32:19Z

internal/discover/symlinks.go

+	if soname == "" || soname == libraryName {
+		return nil, nil
+	}


Do we not still want the .so link even if the SONAME is the same as the libraryName?

Yes, I think you're right. We want the .so link but not the soname link. I don't think this comes up in typical driver installations though, so we may want to skip this for now.

That's fine then. We can always revisit this if anyone complains.

klueska · 2025-07-22T20:33:50Z

internal/discover/symlinks.go

+
+	// Create the .so -> SONAME symlink
+	soLink := getSoLink(soname)
+	if soLink != "" && soLink != soname && soLink != libraryName {


I don't understand the need for all these special cases

If we get "" this means that soname is not a .so.* file which one could argue is unexpected since we do filter on driver libraries at the start of this function. The other cases make sure that we don't try to create a link that links to itself. I think these are unlikely to occur in our driver installations, but are there for completenes.

tests/e2e/Makefile

tests/e2e/nvidia-container-toolkit_test.go

internal/discover/symlinks.go

klueska · 2025-07-23T09:24:02Z

internal/discover/symlinks.go

+	// Create the .so -> SONAME symlink
+	if soLink := getSoLink(soname); soLink != "" && soLink != soname {
+		s := Symlink{
+			target: soname,
+			link:   filepath.Join(libraryDir, soLink),
+		}
+		soSymlinks = append(soSymlinks, s.String())
+	}


I still think this could be a bit more readable (in terms of the conditional cases). Maybe just a comment on what the cases are / mean would be sufficient, e.g.:

// Create the .so -> SONAME symlink // If soname does not have a '.so' extension or soname itself ends with '.so' // we skip this (because we will have just created the same symlink above).

klueska · 2025-07-23T09:24:36Z

A couple of minor comments, otherwise LGTM

This change ensures that .so and SONAME symlinks are created for driver libraries in the container. Signed-off-by: Evan Lezar <[email protected]>

This change removes the create-soname-symlinks hook introduced in v1.18.0-rc.1. Instead we rely on explicitly creating the .so -> SONAME -> .so.RM_VERSION symlink chain through the create-symlink hook. Signed-off-by: Evan Lezar <[email protected]>

Signed-off-by: Evan Lezar <[email protected]>

klueska · 2025-07-23T11:21:52Z

Looking good. Thanks.

ArangoGutierrez

LGTM

elezar self-assigned this Feb 13, 2024

elezar mentioned this pull request Feb 14, 2024

Add support for feature flags #362

Merged

elezar force-pushed the CNT-4766/create-so-symlinks branch 4 times, most recently from e20826f to 7b5f6b3 Compare April 3, 2024 15:10

elezar marked this pull request as ready for review April 3, 2024 15:10

elezar requested review from klueska and cdesiniotis April 3, 2024 15:11

klueska reviewed Apr 15, 2024

View reviewed changes

elezar commented Apr 15, 2024

View reviewed changes

elezar force-pushed the CNT-4766/create-so-symlinks branch 3 times, most recently from 93753a9 to 020f598 Compare April 15, 2024 14:31

elezar requested a review from klueska May 15, 2024 12:32

elezar marked this pull request as draft June 18, 2025 21:42

elezar force-pushed the CNT-4766/create-so-symlinks branch from 020f598 to 81a6b00 Compare July 17, 2025 09:51

klueska reviewed Jul 17, 2025

View reviewed changes

cmd/nvidia-ctk/info/info.go Outdated Show resolved Hide resolved

elezar force-pushed the CNT-4766/create-so-symlinks branch from 81a6b00 to 3a1f96c Compare July 21, 2025 10:01

elezar requested a review from klueska July 21, 2025 10:12

elezar marked this pull request as ready for review July 21, 2025 10:12

elezar added this to the v1.18.0 milestone Jul 21, 2025

ArangoGutierrez requested review from ArangoGutierrez and Copilot July 21, 2025 10:22

elezar force-pushed the CNT-4766/create-so-symlinks branch from 20abfcc to 3fea29c Compare July 21, 2025 10:22

This comment was marked as outdated.

Sign in to view

klueska reviewed Jul 21, 2025

View reviewed changes

pkg/nvcdi/driver-nvml.go Outdated Show resolved Hide resolved

elezar force-pushed the CNT-4766/create-so-symlinks branch 9 times, most recently from e205edd to bc7ed44 Compare July 22, 2025 20:18

klueska reviewed Jul 22, 2025

View reviewed changes

internal/discover/symlinks.go Outdated Show resolved Hide resolved

klueska reviewed Jul 22, 2025

View reviewed changes

ArangoGutierrez requested changes Jul 23, 2025

View reviewed changes

tests/e2e/Makefile Outdated Show resolved Hide resolved

tests/e2e/nvidia-container-toolkit_test.go Outdated Show resolved Hide resolved

klueska reviewed Jul 23, 2025

View reviewed changes

internal/discover/symlinks.go Show resolved Hide resolved

klueska reviewed Jul 23, 2025

View reviewed changes

elezar force-pushed the CNT-4766/create-so-symlinks branch 2 times, most recently from 87082a6 to a560b8b Compare July 23, 2025 10:40

elezar added 6 commits July 23, 2025 13:16

Create .so and SONAME symlinks for driver libraries

e7a1e65

This change ensures that .so and SONAME symlinks are created for driver libraries in the container. Signed-off-by: Evan Lezar <[email protected]>

Remove create-soname-symlinks hook

5b74b44

This change removes the create-soname-symlinks hook introduced in v1.18.0-rc.1. Instead we rely on explicitly creating the .so -> SONAME -> .so.RM_VERSION symlink chain through the create-symlink hook. Signed-off-by: Evan Lezar <[email protected]>

[no-relnote] Don't read ldconfig path from config

37fde66

Signed-off-by: Evan Lezar <[email protected]>

[no-relnote] Allow local tests

0dab248

Signed-off-by: Evan Lezar <[email protected]>

[no-relnote] Remove CUDA repo references

6ec92b5

Signed-off-by: Evan Lezar <[email protected]>

[no-relnote] Fix minor typo in README

3a59233

Signed-off-by: Evan Lezar <[email protected]>

elezar force-pushed the CNT-4766/create-so-symlinks branch from a560b8b to 3a59233 Compare July 23, 2025 11:16

klueska approved these changes Jul 23, 2025

View reviewed changes

ArangoGutierrez approved these changes Jul 23, 2025

View reviewed changes

elezar merged commit db68c39 into NVIDIA:main Jul 23, 2025
16 checks passed

elezar deleted the CNT-4766/create-so-symlinks branch July 23, 2025 12:22

Create .so symlinks for driver libraries in container #326

Create .so symlinks for driver libraries in container #326

Uh oh!

Conversation

elezar commented Feb 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

klueska left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ArangoGutierrez commented Feb 25, 2025

Uh oh!

elezar commented Feb 25, 2025

Uh oh!

coveralls commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 16469131076

Details

💛 - Coveralls

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

klueska Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

elezar Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

klueska Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

klueska Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

elezar Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

klueska Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

klueska commented Jul 23, 2025

Uh oh!

klueska commented Jul 23, 2025

Uh oh!

ArangoGutierrez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elezar commented Feb 1, 2024 •

edited

Loading

coveralls commented Jul 17, 2025 •

edited

Loading