Skip to content

Implemented capability to isolate worker dependencies #794

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jan 13, 2021
Merged

Conversation

Hazhzeng
Copy link
Contributor

@Hazhzeng Hazhzeng commented Dec 14, 2020

Description

Fixes #789

The new feature is implemented under PYTHON_ISOLATE_WORKER_DEPENDENCIES feature flag.

This PR is to introduce a capability to separate customer's site-packages from worker's dependencies.
We create a wrapper for loading the modules and ensure the sys.path and sys.modules have consistent content (invalidate overlapped cache) as the worker begin loading customer's code.

This will permanently address the issue when we bump up protobuf version in the future Python worker when customer is using an old version of protobuf.

When the new flag is set, we should see the following logging entries:

  1. Linux Consumption: Reloaded azure google namespace from customer dependencies
  2. Linux Dedicated: Start using customer dependencies {cls.cx_deps_path}

PR information

  • The title of the PR is clear and informative.
  • There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
  • If applicable, the PR references the bug/issue that it fixes in the description.
  • New Unit tests were added for the changes made and CI is passing.

Quality of Code and Contribution Guidelines

@Hazhzeng Hazhzeng marked this pull request as draft December 14, 2020 20:01
@Hazhzeng Hazhzeng changed the title [WIP] Implemented capability to isolate worker dependencies Implemented capability to isolate worker dependencies Dec 21, 2020
@Hazhzeng Hazhzeng marked this pull request as ready for review December 21, 2020 21:13
@Hazhzeng Hazhzeng force-pushed the hazeng/isolate branch 2 times, most recently from ea329f1 to 3b4c737 Compare December 22, 2020 19:37
@Hazhzeng
Copy link
Contributor Author

I tried many ways to isolate the environment variable changes impact on other tests, but only find separating the testing file works. Let me know if you have any questions on this one.

@Hazhzeng Hazhzeng merged commit 8404b64 into dev Jan 13, 2021
@Hazhzeng Hazhzeng deleted the hazeng/isolate branch January 13, 2021 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'
2 participants