Skip to content

bpo-37210: Fix pure Python pickle: proto 5 requires _pickle.PickleBuffer #14016

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 13, 2019
Merged

bpo-37210: Fix pure Python pickle: proto 5 requires _pickle.PickleBuffer #14016

merged 2 commits into from
Jun 13, 2019

Conversation

vstinner
Copy link
Member

@vstinner vstinner commented Jun 12, 2019 •
edited by bedevere-bot
Loading

Fix pure Python implementation of pickle: it failed with
ModuleNotFoundError on importing _pickle.PickleBuffer. If
_pickle.PickleBuffer is missing, change HIGHEST_PROTOCOL from 5 to 4
and disable PickleBuffer support.

https://bugs.python.org/issue37210

@vstinner vstinner requested a review from pitrou June 12, 2019 14:12
@vstinner
Copy link
Member Author

@pitrou: Would you mind to review my change? Does it sound reasonable to you?

@pitrou
Copy link
Member

pitrou commented Jun 12, 2019

This sounds both fine but almost pointless to me. Is it for performance?

@vstinner
Copy link
Member Author

This sounds both fine but almost pointless to me. Is it for performance?

It repairs pyperformance, right: to measure performances of the pure Python implementation of pickle.

But other Python implementations (PyPy, MicroPython, RustPython, etc.) should also benefit of this change to reuse CPython pickle.py if they miss the _pickle module.

@pitrou
Copy link
Member

pitrou commented Jun 12, 2019

Ok. I'm not sure you need to downgrade the protocol to 4. You can just disable the code paths which depend on PickleBuffer.

@vstinner
bpo-37210: Fix pure Python implementation of pickle
701d3de 
Fix pure Python implementation of pickle: it failed with
ModuleNotFoundError on importing _pickle.PickleBuffer. If
_pickle.PickleBuffer is missing, disable PickleBuffer support.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
@vstinner
Copy link
Member Author

I'm not sure you need to downgrade the protocol to 4. You can just disable the code paths which depend on PickleBuffer.

Oh, I wasn't sure. I tested test_pickle with import sys; sys.modules['_pickle'] = None: I confirm that test_pickle pass if I leave HIGHEST_PROTOCOL unchanged (equal to 5).

I rebased my PR, I reverted my HIGHEST_PROTOCOL change and I made another tiny fix in test_pickle when _pickle is missing.

@vstinner
Copy link
Member Author

@pitrou: Does it look better now?

pitrou
pitrou approved these changes Jun 13, 2019
View reviewed changes
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

@vstinner vstinner merged commit 63ab4ba into python:master Jun 13, 2019
@miss-islington
Copy link
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

@vstinner vstinner deleted the pickle_pure_python branch June 13, 2019 11:58
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 13, 2019
@vstinner @miss-islington
bpo-37210: Fix pure Python pickle when _pickle is unavailable (python…
f1abb42 
…GH-14016)

Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
(cherry picked from commit 63ab4ba)

Co-authored-by: Victor Stinner <[email protected]>
@bedevere-bot
Copy link

GH-14055 is a backport of this pull request to the 3.8 branch.

@vstinner
Copy link
Member Author

@pitrou: Thanks for the review, I updated the commit message using your NEWS entry ;-)

miss-islington added a commit that referenced this pull request Jun 13, 2019
@miss-islington @vstinner
bpo-37210: Fix pure Python pickle when _pickle is unavailable (GH-14016)
cbda40d 
Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
(cherry picked from commit 63ab4ba)

Co-authored-by: Victor Stinner <[email protected]>
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
@vstinner @lisroach
bpo-37210: Fix pure Python pickle when _pickle is unavailable (python…
5752232 
…GH-14016)

Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
@vstinner @DinoV
bpo-37210: Fix pure Python pickle when _pickle is unavailable (python…
5f72a32 
…GH-14016)

Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pitrou pitrou pitrou approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@vstinner @pitrou @miss-islington @bedevere-bot @the-knights-who-say-ni