Skip to content

Python Worker Extension Interface (worker) #815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Apr 1, 2021
Merged

Conversation

Hazhzeng
Copy link
Contributor

@Hazhzeng Hazhzeng commented Feb 3, 2021

Changes

  1. Add ExtensionManager in extension.py which can be enabled by the feature flag PYTHON_ENABLE_WORKER_EXTENSIONS
  2. Add trace_parent, trace_state and attributes replicating TraceParent, TraceState and Attributes values. This change will ensure the worker code is more Pythonic.
  3. Improve logging in worker_init_request, now the log entry will include information like Python version, Worker version for ease of diagnostics.
    Received WorkerInitRequest, python version 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)], worker version 1.1.10, request ID 768c03e6-b194-49ab-9866-0f9978fcdf1d
  4. Add new Python Worker Extension logs for reporting registered Python Worker Extensions.
  5. Add unit test cases in test_extension.py to validate worker extension implementation.

Description

This PR is based on #757

This PR has a dependency on Azure/azure-functions-python-library#81

This extension manager is intended to be used with azure.functions SDK. There're three roles involves in this scenarios.

  1. Python function worker and SDK maintainers
  2. Extension developers
  3. Customers using Python Azure Functions

To enable this feature, two feature flags needs to be turned on in Python 3.6, 3.7 and 3.8.
Python 3.9 enables this feature by default.
PYTHON_ISOLATE_WORKER_DEPENDENCIES this will ensure when registering the extension, it will always direct to customer's azure-functions SDK instead of the built-in one.
PYTHON_ENABLE_WORKER_EXTENSIONS this will ensure the extension loaders are enabled. Otherwise, it will be ignored.

1. Python function worker and SDK maintainers

We decided to release a new extension module in azure.functions SDK where Extension developers can use it to register their extension. (Azure/azure-functions-python-library#81)

2. Extension developers

Here are two examples for developing Application Level Extension and Function Level Extension for Python Worker.
Function Extension: https://github.com/Hazhzeng/functions-ext-profile
Application Extension: https://github.com/Hazhzeng/application-ext-timer

3. Azure Functions Users

The need to download the extension and azure-functions in their requirements.txt

azure-functions
git+https://github.com/Hazhzeng/functions-ext-profile
git+https://github.com/Hazhzeng/application-ext-timer

To use a function extension:

import logging

import requests
import azure.functions as func
from functions_ext_profile import ProfileExtension

profile_extension = ProfileExtension(__file__)


def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Start Executing HttpFuncExt')
    requests.get('https://microsoft.com')
    # profile_extension.property
    return 'HttpFuncExt Executed: called https://microsoft.com'

To apply an application extension into every triggers:

import logging

import requests
import azure.functions as func

# Import application level extension
from application_ext_timer import TimerExtension


def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Start Executing HttpFuncExt')
    requests.get('https://xbox.com')
    return 'HttpAppExt Executed: called https://xbox.com'

Fixes #642
cc: @lzchen


PR information

  • The title of the PR is clear and informative.
  • There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
  • If applicable, the PR references the bug/issue that it fixes in the description.
  • New Unit tests were added for the changes made and CI is passing.

Quality of Code and Contribution Guidelines

@Hazhzeng Hazhzeng marked this pull request as draft February 3, 2021 22:00
@Hazhzeng Hazhzeng marked this pull request as ready for review February 5, 2021 19:47
@Hazhzeng Hazhzeng force-pushed the hazeng/extension-wrk branch from 3e21058 to 52b5c7b Compare February 10, 2021 21:40
@Hazhzeng Hazhzeng force-pushed the hazeng/extension-wrk branch from 206ea5a to 0386a23 Compare March 6, 2021 08:19
@Hazhzeng Hazhzeng changed the title Add extension manager in Python worker Python Worker Extension Interface (worker) Mar 6, 2021
@Hazhzeng Hazhzeng force-pushed the hazeng/extension-wrk branch from 7352a99 to 65f9e66 Compare March 23, 2021 23:12
@Hazhzeng Hazhzeng force-pushed the hazeng/extension-wrk branch from 65f9e66 to cb8624c Compare March 24, 2021 22:59
Copy link
Member

@vrdmr vrdmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An initial set of small comments.

Copy link
Member

@vrdmr vrdmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second set of comments. Still to finish the test_extension.py

Hanzhang Zeng (Roger) added 2 commits March 31, 2021 14:19
Copy link
Member

@vrdmr vrdmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM;

minor comments.

@Hazhzeng Hazhzeng merged commit 7cee871 into dev Apr 1, 2021
@Hazhzeng Hazhzeng deleted the hazeng/extension-wrk branch April 1, 2021 23:12
@lzchen lzchen mentioned this pull request May 5, 2021
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

How to enrich the logger with custom properties?
2 participants