Pip packaging and publishing improvements in pytorch wheels for better integration with poetry

[juliusfrost]

Please see the issues described here: pytorch/pytorch#76557

[jzazo]

I am not sure how or why, but poetry add torch==1.13.1 installs all needed dependencies, but poetry add torch==2.0.0 does not install necessary packages and does not work. Is there a way that works with poetry?

Things work with pip install torch==2.0.0. Thanks.

[vit-zikmund]

I did a little investigation, trying to figure out if this would be feasible for a PR. Here's what I've found what might be done to get this further

Investigation

• The main objective is to add the URL fragment (#<hashname>=<hashvalue>) to the Simple Repository package index.html described in PEP503. The recommended hash algorithm is sha256, so we should aim for that.

• All the published files normally accessible from download.pytorch.org/whl are in fact hosted on this public AWS S3 pytorch bucket and most of the tooling I saw is built around it.

• The tool that generates these indexes sits in this repo as s3_management/manage.py. It does so by scanning select files ("objects" in the S3 jargon) in the bucket, generating the simple html pages from the object keys (that are the fully qualified paths respective to the bucket's root). This script is quite simple to read and modify and adding support for the hashes would surely go to the to_simple_package_html method.

• The main challenge is where to get the file's sha256 digest for the URL Fragment.

• More than a year ago AWS S3 added support for computing/storing additional checksums, that also include the sha256 algorithm. While there is a fairly easy way how to retrieve the checksum for an S3 object using the get_object_attributes call, there's a catch.

• The hash needs to be uploaded along with the content for it to be there. There's a couple ways how to do that, however none involves the locally popular aws s3 cp command. Hopefully, it seems there might be a drop-in replacement using aws s3api put-object described in the issue requesting support in aws s3 cp 🙂

• There are various little scripts doing the uploads to S3 scattered across the repo and for a foreigner like me it's pretty hard to determine what scripts are really used and how. It would be great if some good soul from the local release engineering could give us some hints.

Objectives

• Enable sha256 checksums along with package uploads.
  Proper places need to be identified and the feasibility of the suggested solution verified. Potentially, some other mechanism of passing the checksum would need to be proposed.
• Add checksum retrieval & publishing logic to s3_management/manage.py.
  This is rather easily done once the hashes are in S3.

[vit-zikmund]

Tagging @seemethere & @malfet with a humble request for comments. Thank you!

[thejcannon]

I think the title likely should reflect what this issue truly represents which is PEP 503-supported hashes on certain package URLs