Skip to content

fix(hadoop): Backport HADOOP-18583 & fix OpenSSL native library #1209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 18, 2025

Conversation

lfrancke
Copy link
Member

@lfrancke lfrancke commented Jul 18, 2025

Description

This was reported as SUP-215 internally.

This fixes

Together they mean that the native library (OpenSSL) was not used for crypto work:

stackable@hdfs-namenode-default-0 /stackable/hadoop-3.4.0 $ hadoop checknative -a
2025-07-17 21:30:51,368 WARN  bzip2.Bzip2Factory (Bzip2Factory.java:isNativeBzip2Loaded(64)) - Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
2025-07-17 21:30:51,371 INFO  zlib.ZlibFactory (ZlibFactory.java:loadNativeZLib(59)) - Successfully loaded & initialized native-zlib library
2025-07-17 21:30:51,371 WARN  erasurecode.ErasureCodeNative (ErasureCodeNative.java:<clinit>(55)) - ISA-L support is not available in your platform... using builtin-java codec where applicable
2025-07-17 21:30:51,385 INFO  nativeio.NativeIO (NativeIO.java:isPmdkAvailable(192)) - The native code was built without PMDK support.
2025-07-17 21:30:51,448 WARN  crypto.OpensslCipher (OpensslCipher.java:<clinit>(94)) - Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
	at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
	at org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
	at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop:  true /stackable/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false 
bzip2:   false 
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

With only the symlink we get this:

stackable@5df9fee56041 /stackable/hadoop-3.4.1-stackable0.0.0-dev $ hadoop checknative
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Native library checking:
hadoop:  true /stackable/hadoop-3.4.1-stackable0.0.0-dev/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false
bzip2:   false
openssl: false EVP_CIPHER_CTX_block_size
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

Which is why we also need HADOOP-18583 so now the native library works for openssl:

[stackable@0a733a8678ce hadoop-3.4.1-stackable0.0.0-dev]$ bin/hadoop checknative
2025-07-17 22:10:26,164 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
2025-07-17 22:10:26,166 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2025-07-17 22:10:26,166 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2025-07-17 22:10:26,184 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /build/src/hadoop/hadoop/patchable-work/worktree/3.4.1/hadoop-dist/target/hadoop-3.4.1-stackable0.0.0-dev/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false
bzip2:   false
openssl: true /lib64/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

Definition of Done Checklist

  • Add an entry to the CHANGELOG.md file
  • Integration tests ran successfully

@lfrancke lfrancke force-pushed the fix/hadoop-native-openssl branch from 930dc4f to 1eedce1 Compare July 18, 2025 09:07
@lfrancke lfrancke changed the title Backport HADOOP-18583 & fix OpenSSL native library fix(hadoop): Backport HADOOP-18583 & fix OpenSSL native library Jul 18, 2025
@lfrancke lfrancke force-pushed the fix/hadoop-native-openssl branch from 1eedce1 to cf342c0 Compare July 18, 2025 09:17
@lfrancke lfrancke self-assigned this Jul 18, 2025
@lfrancke lfrancke moved this to Development: In Progress in Stackable Engineering Jul 18, 2025
@lfrancke lfrancke moved this from Development: In Progress to Development: Waiting for Review in Stackable Engineering Jul 18, 2025
@lfrancke lfrancke marked this pull request as ready for review July 18, 2025 11:11
Copy link
Member

@NickLarsenNZ NickLarsenNZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM provided @dervoeti is happy with the patch that came in

@NickLarsenNZ NickLarsenNZ added this pull request to the merge queue Jul 18, 2025
@NickLarsenNZ NickLarsenNZ moved this from Development: Waiting for Review to Development: Done in Stackable Engineering Jul 18, 2025
Merged via the queue into main with commit 7d630e5 Jul 18, 2025
3 checks passed
@NickLarsenNZ NickLarsenNZ deleted the fix/hadoop-native-openssl branch July 18, 2025 11:39
@lfrancke
Copy link
Member Author

lfrancke commented Jul 22, 2025

Release Notes

fix: Previously Hadoop wouldn't find and use the native OpenSSL library to speed up crypto operations. This was fixed by adding a missing symlink and backporting HADOOP-18583.

@lfrancke lfrancke added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jul 22, 2025
@lfrancke lfrancke moved this from Development: Done to Done in Stackable Engineering Jul 22, 2025
razvan pushed a commit that referenced this pull request Jul 22, 2025
* Backport HADOOP-18583 & fix OpenSSL native library

* Update CHANGELOG.md

Co-authored-by: Nick <[email protected]>

---------

Co-authored-by: Nick <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release/25.7.0 release-note Denotes a PR that will be considered when it comes time to generate release notes.
Projects
Development

Successfully merging this pull request may close these issues.

2 participants