Enable concurrent execution of Python functions

[maiqbal11]

I've managed to isolate an issue where a queue trigger function is unable to call an http endpoint from a function with the same app. This behavior manifests when the functions are synchronous or asynchronous.

Repro steps

Provide the steps required to reproduce the problem:

1. The offending code is here: QueueWithHttpCall.zip. There is a function called QueueTriggerPython which pulls items from queues and triggers a GET request to an endpoint (from the IntakeHttpTrigger function).
2. Activate virtual environment and install requirements: pip install -r requirements.txt.
3. Install extensions: func extensions install
4. Configure local.settings.json to point to the storage account that you are using.
5. Run func host start and then add item to the configured queue.

Expected behavior

The queue trigger should be able to successfully call into the HTTP endpoint and return the correct status code as well as print out the log message to the console.

Actual behavior

The queue trigger is activated but hangs when trying to call the HTTP endpoint, getting stuck at the following point:

Executing 'Functions.IntakeHttpTrigger' (Reason='This function was programmatically called via the host APIs.', Id=411d2a47-71de-4ab1-b231-c99a9794a7cb)

This is after the point at which the host calls into the language worker to process the request.

Known workarounds

1. Both QueueTriggerPython and IntakeHttpTrigger are synchronous. Raise the number of workers in dispatcher.py to 2:

   azure-functions-python-worker/azure/functions_worker/dispatcher.py

   Line 56 in 6068092

   max_workers=1)

2. Both QueueTriggerPython and IntakeHttpTrigger are asynchronous. No known workarounds.

Related information

For the sync case, there appears to be some issue with the worker threads which are blocking themselves rather than processing the request from a function in the same app. This would explain why raising the number of thread workers to 2 caused the call to go through (one thread for queue trigger and one for the subsequent http call it makes). For the async case, it might be a manifestation of a similar issue since we are executing in the main event loop rather than using a separate thread pool.

[maiqbal11]

\cc @asavaritayal @1st1 @elprans

[asavaritayal]

@elprans can you investigate this issue?

[asavaritayal]

Also adding @1st1 since this seems to be related to how we're handling the thread pool.

[elprans]

@maiqbal11 Your async case does not work because you are making a blocking http request, which blocks the entire event loop. Use aiohttp to make self-requests and it would work.

As for the sync case, I don't think we can handle this safely. Increasing max_workers to a value greater than 1 opens a huge can of worms, since it makes user code essentially multi-threaded with all the consequences that has.

[maiqbal11]

@elprans Works as expected when using aiohttp. Thanks for the clarification! Since the sync case can't be handled safely, the recommendation would be to use async constructs when sending requests to the same function app.

[maiqbal11]

\cc @fabiocav @paulbatum @brettcannon @zooba

[maiqbal11]

Re-opening as there is some pending discussion on, potentially, raising the number of workers in the thread pool executor.

[maiqbal11]

Based on a conversation with @paulbatum who had a few questions/ideas about changing the thread pool size. The basic question is whether we should trade off some of the safety guarantees of having single-threaded user code in favor of allowing (potentially non-proficient) users to write sync code that can run more concurrently.

For more context, the issue in this thread was raised by an internal customer with concerns about latency in their calls. It turned out that this particular scenario did not work because there was only one worker in the threadpool and so requests from one function in the app to another could not be processed. It is likely that there will be other customers who face this issue as well - with the expectation that their code will be able to run with some multi-threading that we provide. Each of these customer issues could turn into a support cases that we would need to tackle.

Some discussion questions based on this:

1. What are the pitfalls that we run into when allowing multi-threaded code and how likely are these for the average user case? This would unblock basic scenarios like the one noted in this issue. @elprans, Perhaps you can elaborate more on this.

2. Can we expect concurrency by using multi-threaded code even if we do not utilize async/await constructs? I did an experiment on this where I had an HTTP endpoint responding with a 10 second delay to calls (made using Python requests library) from a queue trigger. I ran it for a batch of 4 queue messages with max_workers=1 and max_workers=4. Both cases produced similar performance (~40 seconds to run in total). In the case where max_workers=4, the expectation would be that when the requests.get(url) is called, that the thread waiting for a response would give control to another thread that has not yet been able to send the request. However, that does not seem to be the case as the threads execute in an essentially synchronous fashion: Waiting for the duration of the GET request before another thread gets to execute. I've been unable to find a concrete answer on this but it does look like threads cede control for certain operations (https://stackoverflow.com/questions/14765071/python-and-truly-concurrent-threads). Would really like to hear more thoughts/get clarity on on this.