Fix issue when isolating libraries when file not ready #854

Hazhzeng · 2021-05-28T00:42:35Z

Description

Fixes an issue in Python 3.9 Linux Consumption plan. This occurs when the customer's azure.functions library is not presented as we tried to clear out its cache prematurely. This will result in a longer cold start time since the function host will restart several times to wait for the customer's content before doing specialization. The worst scenario will cause the website goes into a 404 state for a few seconds during the code start.

Result:
Failure Exception:
KeyError: 'azure.functions'
Stack:
   File '/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py', line 482, in _handle__function_environment_reload_request     DependencyManager.reload_azure_google_namespace(
   File '/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/dependency.py', line 155, in reload_azure_google_namespace     cls.reload_all_namespaces_from_customer_deps(cx_working_dir)
   File '/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/dependency.py', line 207, in reload_all_namespaces_from_customer_deps     cls._remove_from_sys_path(cls.worker_deps_path)
   File '/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/dependency.py', line 259, in _remove_from_sys_path     cls._clear_path_importer_cache_and_modules(path)
   File '/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/dependency.py', line 278, in _clear_path_importer_cache_and_modules     cls._remove_module_cache(path)
   File '/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/dependency.py', line 353, in _remove_module_cache     elif getattr(module, '__path__', None) and getattr(
   File '<frozen importlib._bootstrap_external>', line 1255, in __len__
   File '<frozen importlib._bootstrap_external>', line 1234, in _recalculate
   File '<frozen importlib._bootstrap_external>', line 1230, in _get_parent_path

Discussion

When customer's trigger name collided with customer's libraries, should give a warning.

Mitigation

This is currently impacting the customers using Python 3.9 Linux Consumption. Setting the PYTHON_ISOLATE_WORKER_DEPENDENCIES to 0 will mitigate this issue for now.

Validation:

Use pytest --instafail --cov=./azure_functions_worker --cov-report xml --cov-branch .\tests\unittests\test_extension.py .\tests\unittests\test_utilities_dependency.py --randomly-seed=3973973583 to introduce unexpected module loading order. This should fails with azure.functions key not found before this fix.

Also found that improper namespacing in worker nuget artifact:

The __init__.py file should only be placed in https://github.com/Azure/azure-functions-python-library source code. However, this shouldn't be included in a release package.

According to Python namespace specification https://www.python.org/dev/peps/pep-0420/#specification. Quote:

Namespace packages cannot contain an init.py. As a consequence, pkgutil.extend_path and pkg_resources.declare_namespace become obsolete for purposes of namespace package creation. There will be no marker file or directory for specifying a namespace package.

PR information

The title of the PR is clear and informative.
There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
If applicable, the PR references the bug/issue that it fixes in the description.
New Unit tests were added for the changes made and CI is passing.

Quality of Code and Contribution Guidelines

I have read the contribution guidelines.

Adding namespace import scenarios Ensure changes Add unittests Amend test cases Fix an issue when customer defines a module with collided name Fix test_paths_resolution Fix case

Hazhzeng · 2021-06-03T16:51:12Z

Validation:

Without the fix from Azure/azure-functions-python-library#90:

With the fix from Azure/azure-functions-python-library#90:

Tested with the following mesh docker image and command:
run_host_mesh_custom.ps1

.\run_host_mesh_custom.ps1 -ImageName mcr.microsoft.com/azure-functions/mesh:3.0.15828-python3.9 -PythonVersion 3.9

Hazhzeng · 2021-06-03T22:24:32Z

Validation on Linux Consumption Python 3.9 (image: mcr.microsoft.com/azure-functions/mesh:3.0.15828-python3.9)
Without the change, able to reproduce the issue (expect to fail):

With the latest Worker and Library mounted, able to mitigate this issue. Confirmed with multiple runs (expect to succeed):

Hazhzeng · 2021-06-03T22:31:32Z

Validation on Linux Consumption Python 3.8 (image: mcr.microsoft.com/azure-functions/mesh:3.0.15828-python3.8)
Without any changes, feature flag disabled by default (expect to succeed):

With the latest worker and library mounted, feature flag disabled by default (expect to succeed):

Hazhzeng · 2021-06-03T22:42:31Z

Validation on Linux Dedicated Python 3.9 (image: mcr.microsoft.com/azure-functions/python:3.0.15828-python3.9)
With the original image (expect to succeed):

With the latest change (expect to succeed):

Hazhzeng · 2021-06-03T23:04:15Z

Validation on Linux Dedicated Python 3.8 (image: mcr.microsoft.com/azure-functions/python:3.0.15828-python3.8)
With the original image, feature flag is disabled by default. Customer brings grpcio-1.35.0 / Worker has grpcio-1.33.2 (collision detected, function is not able to start):

With the original image, setting the PYTHON_ISOLATE_WORKER_DEPENDENCIES = 1, function app is now working:

With the latest worker update, feature flag is disabled by default. (collision detected, function is not able to start):

With the latest worker update, enable the feature flag, function is recovered:

Hazhzeng · 2021-06-04T20:31:24Z

A new round of test with the __init__.py removed from worker dependencies azure/ folder.
Python 3.9 consumption:

Python 3.9 dedicated:

Python 3.8 consumption:

Python 3.8 dedicated (expect to fail, since collision in [customer grpcio-1.35.0 / worker grpcio-1.33.2] library):

This reverts commit 8cb98ee.

vrdmr

minor comments.

LGTM

vrdmr · 2021-06-04T19:07:29Z

azure_functions_worker/dispatcher.py

@@ -480,7 +483,7 @@ async def _handle__function_environment_reload_request(self, req):
            )

            # Reload azure google namespaces
-            DependencyManager.reload_azure_google_namespace(
+            DependencyManager.reload_customer_libraries(


Remove the # Reload azure google namespaces comment here. Not needed.

vrdmr · 2021-06-04T21:23:43Z

azure_functions_worker/constants.py

@@ -18,6 +18,7 @@

 # Platform Environment Variables
 AZURE_WEBJOBS_SCRIPT_ROOT = "AzureWebJobsScriptRoot"
+CONTAINER_NAME = "CONTAINER_NAME"


Can we have this constant in dependency.py itself? This isn't being used globally.

vrdmr · 2021-06-04T21:25:32Z

azure_functions_worker/utils/dependency.py

+        if cx_working_dir:
+            working_directory: str = os.path.abspath(cx_working_dir)
+        if not working_directory:
+            working_directory = cls.cx_working_dir


Can there ever be a case that cx_working_dir isn't a property in cls?

vrdmr · 2021-06-04T21:26:36Z

azure_functions_worker/utils/dependency.py

+        # The modules defined in customer's working directory should have the
+        # least priority since we uses the new folder structure.
+        # Please check the "Message to customer" section in the following PR:
+        # https://github.com/Azure/azure-functions-python-worker/pull/726


Can we make a pinned issue as well (or check if this is added to Python Dev Docs itself)?

vrdmr · 2021-06-04T21:27:11Z

azure_functions_worker/utils/dependency.py


    @classmethod
-    def reload_azure_google_namespace(cls, cx_working_dir: str):
+    def reload_customer_libraries(cls, cx_working_dir: str):
        """Reload azure and google namespace, this including any modules in


This is not only doing Azure and Google right? Comment should reflect this now.

vrdmr · 2021-06-04T21:28:59Z

pack/templates/nix_env_gen.yml

@@ -12,7 +12,10 @@ steps:
  inputs:
    disableAutoCwd: true
    scriptPath: 'pack/scripts/nix_deps.sh'
+# The following task is disabled since Python 3.9 to fix azure/ namespace
+# https://github.com/Azure/azure-functions-python-worker/pull/854


Will we re-enable this particular task later?

So this is an actual fix for Python 3.9 and afterward.

vrdmr · 2021-06-04T21:29:39Z

tests/endtoend/dependency_functions/report_dependencies/__init__.py

+
+
+def main(req: func.HttpRequest) -> func.HttpResponse:
+    """This function is an HttpTrigger to check if the modules are loaded from


Nice test! :)

vivek-bsn · 2021-08-05T20:26:08Z

@Hazhzeng Currently the mitigation provided works to avoid 404 errors. Any update on when this fix is expected to roll out to the production so that the app setting 'PYTHON_ISOLATE_WORKER_DEPENDENCIES ' can be removed?

vrdmr · 2021-08-05T23:04:56Z

The fix is currently being rolled out and is halfway across all our fleet. It should be completed by mid-next week to all the geo-locations.

vivek-bsn · 2021-08-06T20:07:57Z

Once the fix is deployed, we can remove the setting 'PYTHON_ISOLATE_WORKER_DEPENDENCIES' to 0 right? (or) do we need to retain it in the app settings?

Hazhzeng requested review from AnatoliB, anirudhgarg and vrdmr as code owners May 28, 2021 00:42

Hazhzeng mentioned this pull request Jun 2, 2021

Disable isolate worker dependencies for Python 3.9 #855

Merged

5 tasks

Hazhzeng closed this in #855 Jun 2, 2021

Hazhzeng reopened this Jun 2, 2021

Hanzhang Zeng (Roger) added 2 commits June 2, 2021 22:06

Fix an issue when isolating libraries when file not mounted

abcc481

Adding namespace import scenarios Ensure changes Add unittests Amend test cases Fix an issue when customer defines a module with collided name Fix test_paths_resolution Fix case

Squash commit

8ac6543

Hazhzeng force-pushed the hazeng/fix-azure-functions branch from 6f43ad1 to 8ac6543 Compare June 3, 2021 05:12

Hanzhang Zeng (Roger) added 2 commits June 2, 2021 22:34

Retry webhost resolution

c5590fa

Reload linux consumption environment

77435cb

Hazhzeng mentioned this pull request Jun 3, 2021

Fix an issue where the _thirdparty package is not distributed correctly Azure/azure-functions-python-library#90

Merged

Hanzhang Zeng (Roger) and others added 2 commits June 3, 2021 11:34

Add more logs in depenency

deceb0f

Merge branch 'dev' into hazeng/fix-azure-functions

8fed77c

Hanzhang Zeng (Roger) added 4 commits June 4, 2021 10:28

Refactor prioritize_customers_dependencies

2ae9a73

Remove unnecessary if statements

a0626c9

Fix linting

bc37886

Fix namespace packaging

8cb98ee

Hanzhang Zeng (Roger) added 5 commits June 4, 2021 13:32

Revert "Fix namespace packaging"

ab9ca53

This reverts commit 8cb98ee.

Only fix namespace in Python 3.9

5f56015

Fix tab typo

e0c6a88

Fix naming

37289aa

Should fetch the data from variables

ea2e1f3

vrdmr approved these changes Jun 4, 2021

View reviewed changes

Hazhzeng merged commit 8f3645c into dev Jun 4, 2021

Hazhzeng deleted the hazeng/fix-azure-functions branch June 4, 2021 21:36

yojagad mentioned this pull request Jun 10, 2021

Function App Cold Starts Results in 404 Before Reaching Function Azure/azure-functions-host#7421

Closed

v-anvari mentioned this pull request Jul 7, 2021

Function app return 404 on first call after long time of idle. Azure/azure-functions-host#7522

Closed



		def main(req: func.HttpRequest) -> func.HttpResponse:
		"""This function is an HttpTrigger to check if the modules are loaded from

Fix issue when isolating libraries when file not ready #854

Fix issue when isolating libraries when file not ready #854

Uh oh!

Conversation

Hazhzeng commented May 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Discussion

Mitigation

Validation:

Also found that improper namespacing in worker nuget artifact:

PR information

Quality of Code and Contribution Guidelines

Uh oh!

Hazhzeng commented Jun 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hazhzeng commented Jun 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hazhzeng commented Jun 3, 2021

Uh oh!

Hazhzeng commented Jun 3, 2021

Uh oh!

Hazhzeng commented Jun 3, 2021

Uh oh!

Hazhzeng commented Jun 4, 2021

Uh oh!

vrdmr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vivek-bsn commented Aug 5, 2021

Uh oh!

vrdmr commented Aug 5, 2021

Uh oh!

vivek-bsn commented Aug 6, 2021

Uh oh!

Uh oh!

Hazhzeng commented May 28, 2021 •

edited

Loading

Hazhzeng commented Jun 3, 2021 •

edited

Loading

Hazhzeng commented Jun 3, 2021 •

edited

Loading