Docker builds not working with older versions of libssh2 and EMBEDDED_LIB=0

[tomaxsas]

Directions

Tried to built native packages for Centos 6/7 as per documentation. But script fails to run setup.py script

Bug reports

Steps to reproduce:

1. Run ./ci/docker/build-packages.sh

Expected behaviour: [What was expected to happen.]
RPM's are built

Actual behaviour: [What actually happened.]
Erros when runing setup.py

Additional info: [Include version of libssh2 and any other relevant information.]

Centos7 log:

[root@5c04acabe44e src]# python setup.py install
running install
running bdist_egg
running egg_info
writing ssh2_python.egg-info/PKG-INFO
writing top-level names to ssh2_python.egg-info/top_level.txt
writing dependency_links to ssh2_python.egg-info/dependency_links.txt
reading manifest file 'ssh2_python.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory 'docker'
warning: no previously-included files found matching '.travis.yml'
writing manifest file 'ssh2_python.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
building 'ssh2.fileinfo' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Ilibssh2/include -I/usr/include/python2.7 -c ssh2/fileinfo.c -o build/temp.linux-x86_64-2.7/ssh2/fileinfo.o -O3
ssh2/fileinfo.c:761:3: error: unknown type name 'libssh2_struct_stat'
   libssh2_struct_stat *_stat;
   ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo___cinit__':
ssh2/fileinfo.c:1110:33: error: 'libssh2_struct_stat' undeclared (first use in this function)
         __pyx_v_self->_stat = ((libssh2_struct_stat *)malloc((sizeof(libssh2_struct_stat))));
                                 ^
ssh2/fileinfo.c:1110:33: note: each undeclared identifier is reported only once for each function it appears in
ssh2/fileinfo.c:1110:54: error: expected expression before ')' token
         __pyx_v_self->_stat = ((libssh2_struct_stat *)malloc((sizeof(libssh2_struct_stat))));
                                                      ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_7st_size___get__':
ssh2/fileinfo.c:1296:68: error: request for member 'st_size' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_libssh2_uint64_t(__pyx_v_self->_stat->st_size); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 41, __pyx_L1_error)
                                                                    ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_7st_mode___get__':
ssh2/fileinfo.c:1356:65: error: request for member 'st_mode' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_unsigned_long(__pyx_v_self->_stat->st_mode); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 45, __pyx_L1_error)
                                                                 ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_6st_ino___get__':
ssh2/fileinfo.c:1416:57: error: request for member 'st_ino' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_ino_t(__pyx_v_self->_stat->st_ino); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 50, __pyx_L1_error)
                                                         ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_8st_nlink___get__':
ssh2/fileinfo.c:1476:59: error: request for member 'st_nlink' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_nlink_t(__pyx_v_self->_stat->st_nlink); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 54, __pyx_L1_error)
                                                           ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_6st_uid___get__':
ssh2/fileinfo.c:1536:57: error: request for member 'st_uid' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_uid_t(__pyx_v_self->_stat->st_uid); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 58, __pyx_L1_error)
                                                         ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_6st_gid___get__':
ssh2/fileinfo.c:1596:57: error: request for member 'st_gid' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_gid_t(__pyx_v_self->_stat->st_gid); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 62, __pyx_L1_error)
                                                         ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_7st_rdev___get__':
ssh2/fileinfo.c:1656:57: error: request for member 'st_rdev' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_dev_t(__pyx_v_self->_stat->st_rdev); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 66, __pyx_L1_error)
                                                         ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_10st_blksize___get__':
ssh2/fileinfo.c:1716:61: error: request for member 'st_blksize' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_blksize_t(__pyx_v_self->_stat->st_blksize); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 70, __pyx_L1_error)
                                                             ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_9st_blocks___get__':
ssh2/fileinfo.c:1776:60: error: request for member 'st_blocks' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_blkcnt_t(__pyx_v_self->_stat->st_blocks); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 74, __pyx_L1_error)
                                                            ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_8st_atime___get__':
ssh2/fileinfo.c:1836:58: error: request for member 'st_atim' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_time_t(__pyx_v_self->_stat->st_atime); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 78, __pyx_L1_error)
                                                          ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_8st_mtime___get__':
ssh2/fileinfo.c:1896:58: error: request for member 'st_mtim' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_time_t(__pyx_v_self->_stat->st_mtime); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 82, __pyx_L1_error)
                                                          ^
ssh2/fileinfo.c: In function '__pyx_pf_4ssh2_8fileinfo_8FileInfo_8st_ctime___get__':
ssh2/fileinfo.c:1953:58: error: request for member 'st_ctim' in something not a structure or union
   __pyx_t_1 = __Pyx_PyInt_From_time_t(__pyx_v_self->_stat->st_ctime); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 86, __pyx_L1_error)

[pkittenis]

Hi there,

Thanks for the interest.

The default libssh2 package on Centos6/7 is not new enough to support all features. Run the build with EMBEDDED_LIB=0 environment variable set - see documentation.

Leaving open to update docker script to do this by default.

[tomaxsas]

Sorry. Forgot to mention that it does not help

[tomaxsas]

I did all steps manually (i have Docker experience) to see that log. Also there are issues with this line

ssh2-python/ci/docker/build-packages.sh

Line 17 in 3499209

docker run -v "$(pwd):/src/" "$name" --rpm-dist $dist -s python -t rpm -d libssh2 -d python setup.py

since you do not provide a command to execute, and there is no entrypoint in base image

[pkittenis]

The docker image used has a default run cmd of fpm. What is provided by the line above are arguments to the fpm command. The script runs fine for me.

Will also need to install Cython for EMBEDDED_LIB to be applied, otherwise it uses the pre-generated C files in the repository which were built with EMBEDDED_LIB=1. This should be installed in the docker image as well, will make that change.

In the above example, the command should be:

EMBEDDED_LIB=0 python setup.py install after Cython is installed.

[tomaxsas]

as i said. It does not work. I will make a video to prove it.

[pkittenis]

If you mean a message like this:

container_linux.go:265: starting container process caused "exec: \"--rpm-dist\": executable file not found in $PATH"

That is something environmental or an issue with the image. Get the same on Travis, which is why system package builds were disabled. On my machine:

$ docker run -v "$(pwd):/src/" centos7 --rpm-dist el7 -s python -t rpm -d libssh2 -d python setup.py
{:timestamp=>"2018-05-24T12:40:54.166037+0000", :message=>"Created package", :path=>"python-ssh2-python-0.12.0+12.gc1c01b6.dirty-1.el7.x86_64.rpm"}

Can see the image usage instructions here. If there is an issue with them, best to raise on their github project.

[tomaxsas]

yes, you are not calling fpm command, just passing parameters, as per documentation https://github.com/cdrx/docker-fpm#usage it show you should call fpm
But it not related with compile issues i'm having

[pkittenis]

after Cython is installed.

Do you have cython installed and have run install with EMBEDDED_LIB=0 set?

[tomaxsas]

Cython wasnt there. After installing it, same issue

[pkittenis]

Could you paste the full log please.

Regarding docker image, it looks like it was changed a while back.