From source builds on system libssh2 without Cython installed

[DerekTBrown]

Bug reports

Steps to reproduce:

1. Rebuild the library using the system libssh, with the appropriate flags set (https://ssh2-python.readthedocs.io/en/latest/installation.html#system-library-build)

Expected behaviour: [What was expected to happen.]

1. Expect to compile the library without the need for agent forwarding logic (ie libssh2_channel_request_auth_agent)

Actual behaviour: [What actually happened.]
Yields a C warning and a runtime error:

ssh2/channel.c:6902:22: warning: implicit declaration of function 'libssh2_channel_request_auth_agent' [-Wimplicit-function-declaration]
           __pyx_v_rc = libssh2_channel_request_auth_agent(__pyx_v_self->_channel);

Very clearly, this function still occurs in the generated output:

ssh2-python/ssh2/channel.c

Line 6902 in 2d62505

__pyx_v_rc = libssh2_channel_request_auth_agent(__pyx_v_self->_channel);

[DerekTBrown]

Also being reported by someone else:
https://bugzilla.redhat.com/show_bug.cgi?id=1631978

[DerekTBrown]

Can someone also explain the logic behind hard-forking libssh2 to create the package?

[JoelStienlet]

Hello Derek,
I've been able to run "password_auth.py" after some changes in "channel.c":
At 3 different places I included :
#ifdef HAVE_AGENT_FWD #endif
lines: L1261, L6861->L6962 and L7168
Do you think it will be useful tu make a "pull request"?
I haven't tested the changes very well yet, so it may be useful to perform more extensive testing before including these modifications into a package.

[pkittenis]

From source builds with HAS_AGENT_FWD=0 require cython to be installed - documentation is lacking. Can use that to work around for now.

See also dockerfile for centos7, and pre-built RPMs on releases page which all link against system libssh2.

It would be best to switch HAS_AGENT_FWD to default to off, and the pre-generated C files to not include it, so that from source builds can work with system libssh2 packages. Might remove it entirely until it is merged upstream.

PRs welcome, or can watch this issue.

I'd also point you to the Code of Conduct. Keep discussions respectful and manners matter when asking things of open source projects.

[DerekTBrown]

Hello Derek,
I've been able to run "password_auth.py" after some changes in "channel.c":
At 3 different places I included :
#ifdef HAVE_AGENT_FWD #endif
lines: L1261, L6861->L6962 and L7168
Do you think it will be useful tu make a "pull request"?
I haven't tested the changes very well yet, so it may be useful to perform more extensive testing before including these modifications into a package.

@JoelStienlet those are auto-generated files, so a PR cannot be made against those files directly.

[DerekTBrown]

I think this is more a versioning problem than anything else. Most downstream projects don't get a choice of which version of libssh to build against.

My suggestion would be to;

• Default to EMBEDDED_LIB=0 and HAVE_AGENT_FORWARD=0
• Release multiple versions of the app like ssh2-python and ssh2-python-next with the pre-compiled C.

[DerekTBrown]

I am still having issues even with Cython installed. It seems Cython's cache is very simplistic, and so it is not overwriting the generated C files.

[pkittenis]

The existing C files will need to be deleted. Can always use the pre-built RPMs on releases page.

No, the embedded source code will not be moved to feature branch, nor will a new package be made just for that.

Most downstream projects use binary wheels, which require bundled libssh2. These are already published.

From source builds have a choice to use either bundled or system libssh2. EMBEDDED_LIB=1 means libssh2 >= 1.6.0 features, has nothing to do with linking - per docs.

Adjusting this means re-generating cython files - there is no other option short of dropping support for < 1.6.0. Cython does not have support for custom ifdef in its generated C code.

Which library to build against can be adjusted for from source builds with the standard -I <..> -R <..> -L <..> arguments - see instructions.

[DerekTBrown]

Sorry for the confusion- I made a lot of assumptions that features existed (ie ifdef, bundling cross-packages), which I guess Cython just doesn't support.

The usecase that it seems both @JoelStienlet and I have is to do a from source build against the system version libssh without Cython installed. One option would be to add a deploy script to one of the CI tools to build a tarball with pre-compiled C for EMEBEDDED_LIB=0, HAS_AGENT_FWD=0. Does that seem like a reasonable solution?

[JoelStienlet]

Hello @DerekTBrown ,
I've used the following commands to install from source without editing any files, with Cython installed:
rm ssh2-python/ssh2/*.c
cd ssh2-python
export EMBEDDED_LIB=0; export HAVE_AGENT_FWD=0; python2 setup.py build
su -c 'python2 setup.py install'
Manually deleting the .c files forced cython to rebuild them.

[pkittenis]

Yes, the above is needed to re-generate cython files. The same is done for building on Windows which also needs it.

The approach being taken is to bundle libssh2 source code, not as a submodule, and target upstream master branch by default. See this branch.

This will allow system libssh2 builds, with a new env flag, without cython or re-generating C files, as well as allow the library to be installed from an sdist using a bundled libssh2 (no dependency) and use a more recent version of libssh2 which is needed to connect to more recent servers.

[JoelStienlet]

Hello Pan and Derek,
I noticed that versioneer tries to open
/usr/lib64/python2.7/site-packages/ssh2_python-0.15.0.post9+1.g2d62505.dirty-py2.7-linux-x86_64.egg/.git/HEAD
when the library is loaded (doing an "strace -e trace=file -f" ).

-> Shouldn't "setup.py install" install this file?
I can copy it manually, but it would certainly be more elegant if it were installed automatically from setup.py.

Another point: versioneer executes /usr/bin/git on runtime. The test script still works if I remove /usr/bin/git. But that seems to imply that the runtime behaviour of versioneer may depend on the presence or not of git.
-> The question is: is there an option to define the version statically on install, so that versioneer doesn't attempt to run git on runtime?

Sorry if my questions are not relevant, I've never used versioneer before, I'm discovering the tool.