Python 3.6 container image: First fig.to_image call is very slow (~ 5 minutes)

[stephprobst]

I'm trying to use kaleido on a Python 3.6 container provided by Redhat.

Container repo: https://github.com/sclorg/s2i-python-container/tree/master/3.6
Image: https://catalog.redhat.com/software/containers/rhscl/python-36-rhel7/599328ef5a13466ac6ac36aa

The minimum code sample I use for testing is:

import plotly.graph_objects as go
fig = go.Figure(data=[go.Scatter(y=[1, 3, 2])])
img_bytes = fig.to_image(format="png", width=200, height=200, scale=1)

I create a new, empty container, connect to it and run the code above. The initial call to the to_image method takes ~ 5 minutes to complete, independent of the assigned resources. All subsequent calls are very fast.

It doesn't matter if the container started a minute ago or half an hour ago. The initial call always takes 5 minutes.

Is this behavior by design? Can the initial start-up procedure really take that long or am I missing something?

Thank you.

[jonmmease]

Hi @Djeramon, it is expected that the first call takes longer that subsequent calls because the Kaleido executable is launched as a subprocess on the first call. But, I haven't seen the first call take more than ~5 seconds, so 5-minutes is definitely not expected.

Thanks for letting us know!

[jonmmease]

@Djeramon Could you add instructions for pulling this image? Is it available without a RedHat subscription?

It would be easier to look into if you could reproduce it using a Centos image available from docker hub. https://hub.docker.com/_/centos.

Thanks!

[Makman2]

Since there is not much traction here, I have to second this. Something makes it start up extremely slowly.

Fedora 31, inside venv (latest plotly and kaleido), using write_image API.

When I hit ctrl+c early, I get following traceback (might be of help):

/home/mak/dev/respice/venv/bin/python /home/mak/dev/respice/examples/thesis/buckmotor.py
Traceback (most recent call last):
  File "/home/mak/dev/respice/examples/thesis/buckmotor.py", line 74, in <module>
    fig.write_image('temp.png')
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/plotly/basedatatypes.py", line 3806, in write_image
    return pio.write_image(self, *args, **kwargs)
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/plotly/io/_kaleido.py", line 252, in write_image
    engine=engine,
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/plotly/io/_kaleido.py", line 132, in to_image
    fig_dict, format=format, width=width, height=height, scale=scale
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/kaleido/scopes/plotly.py", line 104, in transform
    figure, format=format, width=width, height=height, scale=scale
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/kaleido/scopes/base.py", line 280, in _perform_transform
    self._ensure_kaleido()
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/kaleido/scopes/base.py", line 182, in _ensure_kaleido
    startup_response_string = self._proc.stdout.readline().decode('utf-8')
KeyboardInterrupt

Process finished with exit code 1

[jonmmease]

Hi @Makman2, does it eventually finish for you after a long time?

If someone could find a way to reproduce this in a public docker container, that would be really helpful.

[jonmmease]

Another thought for debugging:

• Run the write_image command in an ipython session
• After it hangs for a while, stop it with Control-C
• Run the following and report any output that's printed

import plotly.io as pio
scope = pio.kaleido.scope
print(scope._std_error.getvalue().decode())

Thanks!

[Makman2]

@jonmmease thanks for the quick response! Yes it does finish after a long time. After I have stumbled upon this issue I actually let it run through (also in the magnitude of 5mins).

I haven't yet a docker image ready for reproduction, but inserted your snippet into my scripts, and get this (original traceback included from ctrl+c):

INTERRUPTED AT:
Traceback (most recent call last):
  File "replotlify.py", line 167, in showpatch
    fig.write_image(args.output)
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/plotly/basedatatypes.py", line 3806, in write_image
    return pio.write_image(self, *args, **kwargs)
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/plotly/io/_kaleido.py", line 252, in write_image
    engine=engine,
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/plotly/io/_kaleido.py", line 132, in to_image
    fig_dict, format=format, width=width, height=height, scale=scale
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/kaleido/scopes/plotly.py", line 104, in transform
    figure, format=format, width=width, height=height, scale=scale
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/kaleido/scopes/base.py", line 280, in _perform_transform
    self._ensure_kaleido()
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/kaleido/scopes/base.py", line 182, in _ensure_kaleido
    startup_response_string = self._proc.stdout.readline().decode('utf-8')
KeyboardInterrupt


PLOTLY DEBUG INFO:
[0108/204014.216783:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale 
[0108/204014.222121:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale 
[0108/204014.222140:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale 
[0108/204014.236544:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale 

[jonmmease]

OK, thanks for including that debug info. No issues there, so I guess it really is just running super slow.

A reproducible docker image would be super helpful, thanks!

[Makman2]

I was trying to recreate this in docker as well as on my other PC and failed (same environment: Fedora 31, same PyCharm project with venv).
It runs faster, about as long as the normal figure.show() function does take.
I tried to recreate it on the fedora:31, python:3.9 and python:3.7 (default on Fedora 31) images with no success.

There is one major difference though: My research computer (where it takes long) lies behind a proxy server and university networking. Not sure if this might be related, but as some sort of strapped down Chromium is used in Kaleido, it may be? Just an idea.

[jonmmease]

Hi @Makman2, thanks for working on reproducing the issue. If you're using fig.to_image/fig.write_image/plotly.io.to_image/plotly.io.write_image then there shouldn't be any external network interaction going on, so I don't think a proxy/firewall setup would cause this.

Does the computer have any other security software like SELinux running on it? An SELinux issue was investigated some in #37.

[Makman2]

Yes SELinux is enabled by default on Fedora.

[Makman2]

Since I am not able to reproduce right now, I still have access to the failing machine (though only another 2-3 months), is there maybe something I could further investigate from my end? Like running certain debug statements or looking through certain logs?

[jonmmease]

Thanks for offering @Makman2, unfortunately I don't have any other ideas right now. Could you temporarily disable SELinux to see if that makes a difference?

[Makman2]

I tried it, and no difference. Using setenforce 0 to put SELinux into permissive mode.

But since the idea with the proxy did stuck in my head, out of joke I just disabled my whole internet connection and re-run a simple plotly script that saves an image. AND: It got fast immediately!
So apparently there is some outbound connection happening, and maybe due to the proxy, something starts up until a long timeout is reached after ~5 minutes.

Until now I have run all my scripts out of PyCharm, which doesn't configure any proxy variables at all.
From the standard command line, I have configured HTTP_PROXY and friends (by default without credentials, but this does not seem to matter). The scripts run well and fast in such an environment.

Conclusion: If you are behind a proxy, you must configure proxy related environment variables (with or without credentials does not matter it appears, kaleido just needs the proxy address).

I guess to reproduce this, one needs to setup a restrictive proxy that regulates outbound connections, i.e. every simple query must go through the proxy, a bypass is not allowed (so you have to make sure that if the proxy is off, you can't access the internet without it). There appear to be plenty of proxy server implementations available that can be run quickly.