Internal Server Error on pushing to Container Registry when storage type for packages is minio

[dvjn]

Description

I have configured an s3 bucket to store packages.

[storage]
STORAGE_TYPE = minio
SERVE_DIRECT = true
MINIO_ENDPOINT = s3.ap-south-1.amazonaws.com
MINIO_ACCESS_KEY_ID = ***
MINIO_SECRET_ACCESS_KEY = ***
MINIO_BUCKET = ***
MINIO_LOCATION = ap-south-1
MINIO_BASE_PATH = /data/gitea/misc
MINIO_USE_SSL = true

[storage.packages]
MINIO_BASE_PATH = /data/gitea/packages

When I try to push a container:

1. Each layer takes a retry to push

The push refers to repository [***/divya.jain/homer]
d79bd6a8365d: Pushed 
e68e7f72b9a6: Pushed 
0dca13ec4143: Pushed 
ad14e0fb5549: Pushed 
4e0001f6ed71: Pushed 
f3c9055b7fd6: Retrying in 4 seconds 
994393dc58e7: Retrying in 4 seconds 

2. It ends with an internal server error

The push refers to repository [***/divya.jain/homer]
d79bd6a8365d: Pushed 
e68e7f72b9a6: Pushed 
0dca13ec4143: Pushed 
ad14e0fb5549: Pushed 
4e0001f6ed71: Pushed 
f3c9055b7fd6: Pushed 
994393dc58e7: Pushed 
received unexpected HTTP status: 500 Internal Server Error

When I tried to investigate the logs, I found 2 different types of errors for different containers

Type 1 (in container.EndUploadBlob):

2022/10/02 18:33:38 [6339d982] router: completed GET /v2/ for ***:0, 401 Unauthorized in 0.1ms @ container/container.go:113(container.ReqContainerAccess)
2022/10/02 18:33:39 [6339d982-2] router: completed GET /v2/token?account=divya.jain&scope=repository%3Adivya.jain%2Fhomer%3Apush%2Cpull&service=container_registry for ***:0, 200 OK in 95.4ms @ container/container.go:137(container.Authenticate)
2022/10/02 18:33:39 [6339d983-2] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:b9921724b705025663ffedd9ee6058a69630037c7ae4793e8bbfc7ad12727e7a for ***:0, 404 Not Found in 16.7ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:39 [6339d983-5] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:1cfe1c2fcbd63648fd717163e09d641d2556e1a8a4eed8b1b06375db4f9150b2 for ***:0, 404 Not Found in 17.8ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:39 [6339d983-4] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:4b9fb091993f2d3745facb4eaa28865f24e02a88235f03120f02cab2387396ea for ***:0, 404 Not Found in 35.0ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:39 [6339d983-3] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:c2d6d98eb0ad76128702bbaf1c7b1d27800c3ff047bf4c38e2fa63e529fd742f for ***:0, 404 Not Found in 47.0ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:39 [6339d983] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:44e606d5c30ec85bd17cc13a405421639bb47264be32ede155bb5f7f59eb2a4c for ***:0, 404 Not Found in 124.1ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:39 [6339d983-6] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 144.0ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:39 [6339d983-7] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 146.9ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:39 [6339d983-8] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 201.8ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:39 [6339d983-9] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 245.1ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:39 [6339d983-10] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 301.9ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:39 [6339d983-11] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/u5y4g7jhksb0livtd2q71ewyj for ***:0, 202 Accepted in 309.3ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:39 [6339d983-14] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/itaijpzkc1fgq8udcmvko4sds for ***:0, 202 Accepted in 231.7ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:39 [6339d983-13] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/af7mr9wfgbxbdmkkvjbqmdmap for ***:0, 202 Accepted in 346.9ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:39 ...ntainer/container.go:84:apiError() [E] [6339d983-16] close /data/gitea/tmp/package-upload/u5y4g7jhksb0livtd2q71ewyj: file already closed 
2022/10/02 18:33:39 [6339d983-16] router: completed PUT /v2/divya.jain/homer/blobs/uploads/u5y4g7jhksb0livtd2q71ewyj?digest=sha256%3Ab9921724b705025663ffedd9ee6058a69630037c7ae4793e8bbfc7ad12727e7a for ***:0, 500 Internal Server Error in 268.0ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:39 ...ntainer/container.go:84:apiError() [E] [6339d983-18] close /data/gitea/tmp/package-upload/af7mr9wfgbxbdmkkvjbqmdmap: file already closed 
2022/10/02 18:33:39 [6339d983-18] router: completed PUT /v2/divya.jain/homer/blobs/uploads/af7mr9wfgbxbdmkkvjbqmdmap?digest=sha256%3A4b9fb091993f2d3745facb4eaa28865f24e02a88235f03120f02cab2387396ea for ***:0, 500 Internal Server Error in 260.8ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:39 ...ntainer/container.go:84:apiError() [E] [6339d983-17] close /data/gitea/tmp/package-upload/itaijpzkc1fgq8udcmvko4sds: file already closed 
2022/10/02 18:33:39 [6339d983-17] router: completed PUT /v2/divya.jain/homer/blobs/uploads/itaijpzkc1fgq8udcmvko4sds?digest=sha256%3Ac2d6d98eb0ad76128702bbaf1c7b1d27800c3ff047bf4c38e2fa63e529fd742f for ***:0, 500 Internal Server Error in 423.1ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:40 [6339d983-12] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/jovlqnu1hlfn06neqfksq1v3p for ***:0, 202 Accepted in 920.3ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:40 [6339d983-15] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/ctxbod81ws65p7fqgjakmisec for ***:0, 202 Accepted in 689.9ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:40 ...ntainer/container.go:84:apiError() [E] [6339d984] close /data/gitea/tmp/package-upload/jovlqnu1hlfn06neqfksq1v3p: file already closed    
2022/10/02 18:33:40 [6339d984] router: completed PUT /v2/divya.jain/homer/blobs/uploads/jovlqnu1hlfn06neqfksq1v3p?digest=sha256%3A1cfe1c2fcbd63648fd717163e09d641d2556e1a8a4eed8b1b06375db4f9150b2 for ***:0, 500 Internal Server Error in 183.1ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:40 ...ntainer/container.go:84:apiError() [E] [6339d984-2] close /data/gitea/tmp/package-upload/ctxbod81ws65p7fqgjakmisec: file already closed  
2022/10/02 18:33:40 [6339d984-2] router: completed PUT /v2/divya.jain/homer/blobs/uploads/ctxbod81ws65p7fqgjakmisec?digest=sha256%3A44e606d5c30ec85bd17cc13a405421639bb47264be32ede155bb5f7f59eb2a4c for ***:0, 500 Internal Server Error in 317.3ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:44 [6339d988] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 82.2ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:44 [6339d988-2] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/zfgbxqngsz7ajlu9dksd89uep for ***:0, 202 Accepted in 90.0ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:45 [6339d988-3] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 251.3ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:45 [6339d988-4] router: completed PUT /v2/divya.jain/homer/blobs/uploads/zfgbxqngsz7ajlu9dksd89uep?digest=sha256%3Ab9921724b705025663ffedd9ee6058a69630037c7ae4793e8bbfc7ad12727e7a for ***:0, 201 Created in 188.4ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:45 [6339d989] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 217.7ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:45 [6339d989-2] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/adpflbt5kxfjqcturwe5laurk for ***:0, 202 Accepted in 186.0ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:45 [6339d989-3] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:b9921724b705025663ffedd9ee6058a69630037c7ae4793e8bbfc7ad12727e7a for ***:0, 200 OK in 221.9ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:45 [6339d989-4] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/3fwuvvivxx2exy7iqjnmnojny for ***:0, 202 Accepted in 172.3ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:45 [6339d989-5] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 170.5ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:45 [6339d989-7] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:ea9309e8668568f3c1138d9df7b535a98f7f9d37a22f01824c8f1674ae17b662 for ***:0, 404 Not Found in 85.1ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:45 [6339d989-9] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 105.7ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:45 [6339d989-10] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/0lwzdpctgy03htwxhqqouy4ir for ***:0, 202 Accepted in 178.7ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:45 [6339d989-6] router: completed PUT /v2/divya.jain/homer/blobs/uploads/adpflbt5kxfjqcturwe5laurk?digest=sha256%3A4b9fb091993f2d3745facb4eaa28865f24e02a88235f03120f02cab2387396ea for ***:0, 201 Created in 407.7ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:45 [6339d989-8] router: completed PUT /v2/divya.jain/homer/blobs/uploads/3fwuvvivxx2exy7iqjnmnojny?digest=sha256%3Ac2d6d98eb0ad76128702bbaf1c7b1d27800c3ff047bf4c38e2fa63e529fd742f for ***:0, 201 Created in 410.9ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:45 [6339d989-11] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 369.8ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:45 [6339d989-14] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:4b9fb091993f2d3745facb4eaa28865f24e02a88235f03120f02cab2387396ea for 
***:0, 200 OK in 130.6ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:45 [6339d989-12] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/iueuhhkeh652bfrnboszswltk for ***:0, 202 Accepted in 399.6ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:45 [6339d989-17] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49 for 
***:0, 404 Not Found in 74.6ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:46 [6339d989-15] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:c2d6d98eb0ad76128702bbaf1c7b1d27800c3ff047bf4c38e2fa63e529fd742f for 
***:0, 200 OK in 197.2ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:46 [6339d989-13] router: completed PUT /v2/divya.jain/homer/blobs/uploads/0lwzdpctgy03htwxhqqouy4ir?digest=sha256%3A1cfe1c2fcbd63648fd717163e09d641d2556e1a8a4eed8b1b06375db4f9150b2 for ***:0, 201 Created in 412.6ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:46 [6339d98a-2] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 142.8ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:46 [6339d98a-3] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:1cfe1c2fcbd63648fd717163e09d641d2556e1a8a4eed8b1b06375db4f9150b2 for ***:0, 200 OK in 80.4ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:46 [6339d98a] router: completed PUT /v2/divya.jain/homer/blobs/uploads/iueuhhkeh652bfrnboszswltk?digest=sha256%3A44e606d5c30ec85bd17cc13a405421639bb47264be32ede155bb5f7f59eb2a4c for ***:0, 201 Created in 243.6ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:46 [6339d989-16] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/vhkbqbjwbq9fnyihy5ydvxwjt for ***:0, 202 Accepted in 468.7ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:46 [6339d98a-5] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:44e606d5c30ec85bd17cc13a405421639bb47264be32ede155bb5f7f59eb2a4c for ***:0, 200 OK in 138.4ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:46 ...ntainer/container.go:84:apiError() [E] [6339d98a-6] close /data/gitea/tmp/package-upload/vhkbqbjwbq9fnyihy5ydvxwjt: file already closed  
2022/10/02 18:33:46 [6339d98a-6] router: completed PUT /v2/divya.jain/homer/blobs/uploads/vhkbqbjwbq9fnyihy5ydvxwjt?digest=sha256%3Aea9309e8668568f3c1138d9df7b535a98f7f9d37a22f01824c8f1674ae17b662 for ***:0, 500 Internal Server Error in 275.2ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:46 [6339d98a-4] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/pcnob23a7qm5fzskq1x7q8cps for ***:0, 202 Accepted in 570.2ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:46 ...ntainer/container.go:84:apiError() [E] [6339d98a-7] close /data/gitea/tmp/package-upload/pcnob23a7qm5fzskq1x7q8cps: file already closed  
2022/10/02 18:33:46 [6339d98a-7] router: completed PUT /v2/divya.jain/homer/blobs/uploads/pcnob23a7qm5fzskq1x7q8cps?digest=sha256%3A213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49 for ***:0, 500 Internal Server Error in 223.0ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:50 [6339d98e] router: completed GET / for 10.0.0.220:44548, 303 See Other in 0.7ms @ web/home.go:26(web.Home)
2022/10/02 18:33:50 [6339d98e-2] router: completed GET / for 10.0.0.4:58942, 303 See Other in 0.6ms @ web/home.go:26(web.Home)
2022/10/02 18:33:51 [6339d98f] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 87.8ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:52 [6339d98f-3] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 81.4ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:52 [6339d98f-2] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/erfntmtoaauh4rjz90ioifdmo for ***:0, 202 Accepted in 409.8ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:52 [6339d990-2] router: completed PUT /v2/divya.jain/homer/blobs/uploads/erfntmtoaauh4rjz90ioifdmo?digest=sha256%3Aea9309e8668568f3c1138d9df7b535a98f7f9d37a22f01824c8f1674ae17b662 for ***:0, 201 Created in 106.0ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:52 [6339d990-3] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:ea9309e8668568f3c1138d9df7b535a98f7f9d37a22f01824c8f1674ae17b662 for ***:0, 200 OK in 75.4ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:52 [6339d990] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/fdebhsree52kmconxf9gzafi7 for ***:0, 202 Accepted in 540.3ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:52 [6339d990-4] router: completed PUT /v2/divya.jain/homer/blobs/uploads/fdebhsree52kmconxf9gzafi7?digest=sha256%3A213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49 for ***:0, 201 Created in 128.4ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:33:52 [6339d990-5] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49 for ***:0, 200 OK in 67.7ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:52 [6339d990-6] router: completed HEAD /v2/divya.jain/homer/blobs/sha256:13ea515c1e29df3db03803c690e76ccce343744889ceae2b9f21f0d3801b1eb6 for ***:0, 404 Not Found in 39.4ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:33:52 [6339d990-7] router: completed POST /v2/divya.jain/homer/blobs/uploads/ for ***:0, 202 Accepted in 80.6ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:33:53 [6339d990-8] router: completed PATCH /v2/divya.jain/homer/blobs/uploads/ec3y6dwghbvgzubgv7whbdiy5 for ***:0, 202 Accepted in 105.4ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:33:53 ...ntainer/container.go:84:apiError() [E] [6339d991] close /data/gitea/tmp/package-upload/ec3y6dwghbvgzubgv7whbdiy5: file already closed    
2022/10/02 18:33:53 [6339d991] router: completed PUT /v2/divya.jain/homer/blobs/uploads/ec3y6dwghbvgzubgv7whbdiy5?digest=sha256%3A13ea515c1e29df3db03803c690e76ccce343744889ceae2b9f21f0d3801b1eb6 for ***:0, 500 Internal Server Error in 175.1ms @ container/container.go:297(container.EndUploadBlob)

Type 2 (in container.UploadManifest):

2022/10/02 18:43:44 [6339dbe0] router: completed GET /v2/ for ***:0, 401 Unauthorized in 0.1ms @ container/container.go:113(container.ReqContainerAccess)
2022/10/02 18:43:44 [6339dbe0-2] router: completed GET /v2/token?account=divya.jain&scope=repository%3Adivya.jain%2Fplugins-gitea-release%3Apush%2Cpull&service=container_registry for ***:0, 200 OK in 97.2ms @ container/container.go:137(container.Authenticate)
2022/10/02 18:43:44 [6339dbe0-3] router: completed HEAD /v2/divya.jain/plugins-gitea-release/blobs/sha256:2fe460a2a5a17b934b9a339dbf7c155ac69414bd4e5bb9d35770b7dfe695d317 for ***:0, 404 Not Found in 41.0ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:43:44 [6339dbe0-5] router: completed HEAD /v2/divya.jain/plugins-gitea-release/blobs/sha256:df31f910d6917e56e80da012bd8f369817b028ab514a0704fec022a1f9f494cb for ***:0, 404 Not Found in 56.1ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:43:44 [6339dbe0-4] router: completed HEAD /v2/divya.jain/plugins-gitea-release/blobs/sha256:1b3c38b45ba2790f5f260b583a29a44f87770b51c81cf9f75afcbae5a184ff04 for ***:0, 404 Not Found in 78.9ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:43:44 [6339dbe0-6] router: completed GET /v2/token?account=divya.jain&scope=repository%3Adivya.jain%2Fplugins-gitea-release%3Apush%2Cpull&scope=repository%3Adivya.jain%2Ftest-repo%3Apull&service=container_registry for ***:0, 200 OK in 101.8ms @ container/container.go:137(container.Authenticate) 
2022/10/02 18:43:44 [6339dbe0-7] router: completed POST /v2/divya.jain/plugins-gitea-release/blobs/uploads/?from=divya.jain%2Ftest-repo&mount=sha256%3A2fe460a2a5a17b934b9a339dbf7c155ac69414bd4e5bb9d35770b7dfe695d317 for ***:0, 202 Accepted in 106.8ms @ container/container.go:190(container.InitiateUploadBlob) 
2022/10/02 18:43:44 [6339dbe0-8] router: completed GET /v2/token?account=divya.jain&scope=repository%3Adivya.jain%2Fplugins-gitea-release%3Apush%2Cpull&scope=repository%3Adivya.jain%2Ftest-repo%3Apull&service=container_registry for ***:0, 200 OK in 176.1ms @ container/container.go:137(container.Authenticate) 
2022/10/02 18:43:44 [6339dbe0-9] router: completed POST /v2/divya.jain/plugins-gitea-release/blobs/uploads/?from=divya.jain%2Ftest-repo&mount=sha256%3Adf31f910d6917e56e80da012bd8f369817b028ab514a0704fec022a1f9f494cb for ***:0, 202 Accepted in 98.3ms @ container/container.go:190(container.InitiateUploadBlob)  
2022/10/02 18:43:44 [6339dbe0-10] router: completed GET /v2/token?account=divya.jain&scope=repository%3Adivya.jain%2Fplugins-gitea-release%3Apush%2Cpull&scope=repository%3Adivya.jain%2Ftest-repo%3Apull&service=container_registry for ***:0, 200 OK in 166.7ms @ container/container.go:137(container.Authenticate)2022/10/02 18:43:45 [6339dbe0-12] router: completed POST /v2/divya.jain/plugins-gitea-release/blobs/uploads/?from=divya.jain%2Ftest-repo&mount=sha256%3A1b3c38b45ba2790f5f260b583a29a44f87770b51c81cf9f75afcbae5a184ff04 for ***:0, 202 Accepted in 84.5ms @ container/container.go:190(container.InitiateUploadBlob) 
2022/10/02 18:43:45 [6339dbe0-11] router: completed PATCH /v2/divya.jain/plugins-gitea-release/blobs/uploads/v5ekmb2kov8z4pfxlumz19aii for ***:0, 202 
Accepted in 192.8ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:43:45 [6339dbe0-13] router: completed GET /v2/token?account=divya.jain&scope=repository%3Adivya.jain%2Fplugins-gitea-release%3Apush%2Cpull&scope=repository%3Aseclore-admin%2Ftrp%3Apull&service=container_registry for ***:0, 200 OK in 230.9ms @ container/container.go:137(container.Authenticate)   
2022/10/02 18:43:45 [6339dbe1] router: completed POST /v2/divya.jain/plugins-gitea-release/blobs/uploads/?from=seclore-admin%2Ftrp&mount=sha256%3Adf31f910d6917e56e80da012bd8f369817b028ab514a0704fec022a1f9f494cb for ***:0, 202 Accepted in 108.0ms @ container/container.go:190(container.InitiateUploadBlob)      
2022/10/02 18:43:45 [6339dbe1-2] router: completed GET /v2/token?account=divya.jain&scope=repository%3Adivya.jain%2Fplugins-gitea-release%3Apush%2Cpull&scope=repository%3Aseclore-admin%2Ftrp%3Apull&service=container_registry for ***:0, 200 OK in 178.9ms @ container/container.go:137(container.Authenticate)    
2022/10/02 18:43:45 [6339dbe1-4] router: completed POST /v2/divya.jain/plugins-gitea-release/blobs/uploads/?from=seclore-admin%2Ftrp&mount=sha256%3A1b3c38b45ba2790f5f260b583a29a44f87770b51c81cf9f75afcbae5a184ff04 for ***:0, 202 Accepted in 56.2ms @ container/container.go:190(container.InitiateUploadBlob)     
2022/10/02 18:43:45 [6339dbe1-3] router: completed DELETE /v2/divya.jain/plugins-gitea-release/blobs/uploads/bejuguo259ikm0pk8kvpdlgaw for ***:0, 404 
Not Found in 71.4ms @ packages/api.go:292(packages.ContainerRoutes.func2.2)
2022/10/02 18:43:45 [6339dbe1-6] router: completed DELETE /v2/divya.jain/plugins-gitea-release/blobs/uploads/fthsy8hats3omtfpv1mgwyhj9 for ***:0, 404 
Not Found in 26.1ms @ packages/api.go:292(packages.ContainerRoutes.func2.2)
2022/10/02 18:43:45 [6339dbe1-5] router: completed PUT /v2/divya.jain/plugins-gitea-release/blobs/uploads/v5ekmb2kov8z4pfxlumz19aii?digest=sha256%3A2fe460a2a5a17b934b9a339dbf7c155ac69414bd4e5bb9d35770b7dfe695d317 for ***:0, 201 Created in 209.5ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:43:45 [6339dbe1-9] router: completed HEAD /v2/divya.jain/plugins-gitea-release/blobs/sha256:2fe460a2a5a17b934b9a339dbf7c155ac69414bd4e5bb9d35770b7dfe695d317 for ***:0, 200 OK in 14.9ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:43:45 [6339dbe1-7] router: completed PATCH /v2/divya.jain/plugins-gitea-release/blobs/uploads/binee7wsxiksm0tpgvwkzostr for ***:0, 202 Accepted in 247.4ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:43:45 [6339dbe1-10] router: completed PUT /v2/divya.jain/plugins-gitea-release/blobs/uploads/binee7wsxiksm0tpgvwkzostr?digest=sha256%3Adf31f910d6917e56e80da012bd8f369817b028ab514a0704fec022a1f9f494cb for ***:0, 201 Created in 177.2ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:43:45 [6339dbe1-11] router: completed HEAD /v2/divya.jain/plugins-gitea-release/blobs/sha256:df31f910d6917e56e80da012bd8f369817b028ab514a0704fec022a1f9f494cb for ***:0, 200 OK in 66.6ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:43:46 [6339dbe1-8] router: completed PATCH /v2/divya.jain/plugins-gitea-release/blobs/uploads/xqstbjn9zlzbowfhqgqhgxf8b for ***:0, 202 Accepted in 1081.3ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:43:46 [6339dbe2] router: completed PUT /v2/divya.jain/plugins-gitea-release/blobs/uploads/xqstbjn9zlzbowfhqgqhgxf8b?digest=sha256%3A1b3c38b45ba2790f5f260b583a29a44f87770b51c81cf9f75afcbae5a184ff04 for ***:0, 201 Created in 156.0ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:43:46 [6339dbe2-2] router: completed HEAD /v2/divya.jain/plugins-gitea-release/blobs/sha256:1b3c38b45ba2790f5f260b583a29a44f87770b51c81cf9f75afcbae5a184ff04 for ***:0, 200 OK in 71.2ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:43:46 [6339dbe2-3] router: completed HEAD /v2/divya.jain/plugins-gitea-release/blobs/sha256:a5051cea39fab804969e46c21fefc4f8c75b4bfd29c9ef2da37407a97f5d9dbc for ***:0, 404 Not Found in 46.9ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:43:46 [6339dbe2-4] router: completed POST /v2/divya.jain/plugins-gitea-release/blobs/uploads/ for ***:0, 202 Accepted in 81.5ms @ container/container.go:190(container.InitiateUploadBlob)
2022/10/02 18:43:47 [6339dbe2-5] router: completed PATCH /v2/divya.jain/plugins-gitea-release/blobs/uploads/j0oal96ubzybn8eandfoskmdm for ***:0, 202 Accepted in 109.6ms @ container/container.go:252(container.UploadBlob)
2022/10/02 18:43:47 [6339dbe3] router: completed PUT /v2/divya.jain/plugins-gitea-release/blobs/uploads/j0oal96ubzybn8eandfoskmdm?digest=sha256%3Aa5051cea39fab804969e46c21fefc4f8c75b4bfd29c9ef2da37407a97f5d9dbc for ***:0, 201 Created in 154.4ms @ container/container.go:297(container.EndUploadBlob)
2022/10/02 18:43:47 [6339dbe3-2] router: completed HEAD /v2/divya.jain/plugins-gitea-release/blobs/sha256:a5051cea39fab804969e46c21fefc4f8c75b4bfd29c9ef2da37407a97f5d9dbc for ***:0, 200 OK in 77.1ms @ container/container.go:372(container.HeadBlob)
2022/10/02 18:43:47 ...ntainer/container.go:84:apiError() [E] [6339dbe3-3] EOF
2022/10/02 18:43:47 [6339dbe3-3] router: completed PUT /v2/divya.jain/plugins-gitea-release/manifests/latest for ***:0, 500 Internal Server Error in 84.8ms @ container/container.go:440(container.UploadManifest)

All these problems disappear as soon as I switch to local storage.

Gitea Version

1.17

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Official Gitea Docker Image

Database

SQLite

[KN4CK3R]

The close /data/gitea/tmp/package-upload/ec3y6dwghbvgzubgv7whbdiy5: file already closed errors are strange because the path is the local temporary upload copy. There should be no difference when using another storage type.

Do you get this error every time? Do all seven layers fail or just some of them?

[dvjn]

@KN4CK3R all layers are failing but only for certain images, some images are being pushed without a problem.

[KN4CK3R]

Could you test if it's the same problem as in #19586 (comment) Is one of the failing images public so I could test it?

[dvjn]

Could you test if it's the same problem as in #19586 (comment)

Tested it, this is not the same problem. It is still retrying, now just one layer at a time.

Is one of the failing images public so I could test it?

Yes, I've just retagged existing public images to test it.

• plugins/gitea-release:latest gives me the container.UploadManifest error
• containrrr/watchtower:latest gives me the container.EndUploadBlob error

[KN4CK3R]

Great, will test it tomorrow

[KN4CK3R]

I can't reproduce it. Which docker cli version do you use? Your client makes multiple calls to the authenticate endpoint which my client does not. That's not the reason for the error but we use different clients then.

[dvjn]

I can't reproduce it.

:(

Which docker cli version do you use?

Docker version 20.10.17, build 100c701

[KN4CK3R]

I have the same version. Wondering why your client behaves differently. My client does not send the mount (POST /v2/divya.jain/plugins-gitea-release/blobs/uploads/?from=divya.jain%2Ftest-repo&mount=sha256:2fe460a2a5a17b934b9a339dbf7c155ac69414bd4e5bb9d35770b7dfe695d317) or upload delete (DELETE /v2/divya.jain/plugins-gitea-release/blobs/uploads/bejuguo259ikm0pk8kvpdlgaw) requests.

Anyway, the EOF error should be a missing config layer. For plugins/gitea-release:latest the digest is a5051cea39fab804969e46c21fefc4f8c75b4bfd29c9ef2da37407a97f5d9dbc. Could you please check this file is present in your storage? The path should be /data/gitea/packages/a5/05/a5051cea39fab804969e46c21fefc4f8c75b4bfd29c9ef2da37407a97f5d9dbc. There is a valid upload log for this file (PUT /v2/divya.jain/plugins-gitea-release/blobs/uploads/j0oal96ubzybn8eandfoskmdm?digest=sha256:a5051cea39fab804969e46c21fefc4f8c75b4bfd29c9ef2da37407a97f5d9dbc for ***:0, 201 Created) but better check it. The content should be a valid JSON document.

[dvjn]

Yes I checked the file, it is a valid JSON document.

[dvjn]

Can the mount and delete requests have something to do with me already having tried to push the same image multiple times unsuccessfully?

[KN4CK3R]

Maybe, but that should not be a problem. From the log, the mount is ignored and a new upload is instantiated. Do you have the same errors if you use a (local) minio server instance instead of AWS? I have no idea how the EOF error could occur if all 4 files got uploaded. I could create a PR which adds more logging but I'm not sure if this helps in this case.

[dvjn]

Will test that as soon as I get a chance