After upgrading to 4.1.737 - Write-S3Object causes Access Denied

[httran13]

Describe the bug

Hi, after upgrading to 4.1.737 my code for Write-S3Object causes Access Denied. Im not exactly sure why but I've confirmed this by downgrading the module to 4.1.736.0 and my script worked again.

I see there are some changes to checksum but from what I can tell, im not utilizing any multi-part upload. Infact its just a single xml file of kb size

            Write-S3Object -EndpointUrl $Env:S3_ENDPOINT -Region $Env:S3_REGION -ProfileName "my-s3-api" -BucketName $Env:S3_RESULT_BUCKET -Folder "./results/" -KeyPrefix "latest" -SearchPattern "$topFolder/*.xml"

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Expected the Write-S3Object to write object to bucket successfully

Current Behavior

Throws exception with Access Denied when using version 4.1.737

Reproduction Steps

Install previous version
Install-AWSToolsModule AWS.Tools.S3 -CleanUp -Force -RequiredVersion "4.1.736.0"
Run cmd

            Write-S3Object -EndpointUrl $Env:S3_ENDPOINT -Region $Env:S3_REGION -ProfileName "my-s3-api" -BucketName $Env:S3_RESULT_BUCKET -Folder "./results/" -KeyPrefix "latest" -SearchPattern "$topFolder/*.xml"

SHould be success
Install-AWSToolsModule AWS.Tools.S3 -CleanUp -Force -RequiredVersion "4.1.737.0"
run cmd above again

Possible Solution

No response

Additional Information/Context

We are targeting IBM Cloud Object Storage API's, not Amazon

AWS Tools for PowerShell version used

4.1.737.0

PowerShell version used

Name Value

---

PSVersion 7.4.2
PSEdition Core
GitCommitId 7.4.2
OS Darwin 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:03:15 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6000
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

Operating System and version

OSX Sequioa

[dscpinheiro]

The release we had yesterday also impacts non-multipart uploads (we'll calculate a checksum for individual PutObject calls too).

Although not recommended, you can disable the validation by setting an option in the ClientConfig parameter:

$config = New-Object Amazon.S3.AmazonS3Config
$config.RequestChecksumCalculation = 'WHEN_REQUIRED'
Write-S3Object -Bucket '...' -File '...' -Key '...' -ClientConfig $config

Does that work for you? I'm wondering if IBM Cloud Object Storage API doesn't support checksums, which happened for a couple of other 3rd party implementations as well.

[ashishdhingra]

@httran13 Additionally, please refer #370. It also has a Disclaimer about the new data integrity protections for 3rd party services. You could disable Checksum calculation and validation using code below:

$config = New-Object Amazon.S3.AmazonS3Config
$config.RequestChecksumCalculation = 'WHEN_REQUIRED'

Write-S3Object -Bucket '...' -File '...' -Key '...' -ClientConfig $config

Thanks,
Ashish

[staxel]

We experience kind of the same issue:
When running "Get-S3Bucket" using version 4.1.737 and later, I receive the following error message: "The request signature we calculated does not match the signature you provided. Check your key and signing method."
After downgrading to version 4.1.736 it works perfectly fine; so the provided credentials should be correct.

I tried to implement your suggested solution with the RequestChecksumCalculation but this does not change the behavior.

We are also using a 3rd party S3 provider: Wasabi.

[dscpinheiro]

@staxel The when_supported / when_required options are only applicable for operations that are modeled requesting a checksum. ListBuckets (which Get-S3Bucket calls behind the scenes) is not of those operations (and that didn't change in version 4.1.737).

We haven't made any changes to the signing logic either, but one thing that's different is that we'll include extra metadata in the user agent.

Before:

GET
/

host:s3.us-west-2.amazonaws.com
user-agent:AWSPowerShell.Common/4.1.715.0 ua/2.0 os/linux#6.1.119.129 md/ARCH#X64 lang/.NET_Core#8.0.10 exec-env/CloudShell md/aws-sdk-dotnet-core#3.7.400.61 api/S3#3.7.410.3 md/PowerShellCore/7.-1 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20250120T162558Z

After:

GET
/

host:s3.us-west-2.amazonaws.com
user-agent:AWSPowerShell.Common/4.1.739.0 ua/2.0 os/linux#6.1.119.129 md/ARCH#X64 lang/.NET_Core#8.0.10 exec-env/CloudShell md/aws-sdk-dotnet-core#3.7.401.2 api/S3#3.7.412.2 md/PowerShellCore/7.-1 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1 m/Z,b
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20250120T164849Z

Note there's an extra m/Z,b at the end of the user-agent; that is the encoded information of which RequestChecksumCalculation / ResponseChecksumCalculation customers are using (this is so we can keep track of the new feature - the logic is here: https://github.com/aws/aws-sdk-net/blob/main/sdk/src/Services/S3/Custom/Internal/AmazonS3UserAgentHandler.cs).

I don't have access to Wasabi, but is it possible they're not handling the user agent change during their signature validation?

[staxel]

I am already in contact with Wasabi and will keep you updated.