tmp folder not appearing

[pdf9]

Hi, I've been desperately searching for something like this and would love to make use of it for a small project I'm spending way too much time on. I followed the insanely simple setup instructions :) but can't get the tmp folder to appear in my s3 bucket, so I'm assuming I could not get it to work. I want this to work so bad, please help!

Here's my AWS settings - everything else is just like in the setup instructions, forms use clearable file input, etc. I can share repo with you if that'd help. Is just a simple app to upload photos and videos that uses a calendar to track them, am going to give to my family and friends. Problem is it takes forever to upload multiple images at once (one of the main reasons i'm so pumped about using this) and videos though and I'm afraid they won't use due this, and also it's on heroku so the 30 second timeout is a problem! I'll stop rambling...

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_ACCESS_KEY_ID = 'my key'
AWS_SECRET_ACCESS_KEY = 'secret access key'
AWS_STORAGE_BUCKET_NAME = 'bucket name'
AWS_S3_FILE_OVERWRITE = False
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
AWS_DEFAULT_ACL = None
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_LOCATION = 'static'
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}
AWS_STORAGE_BUCKET_NAME_MEDIA = 'media bucket name'
AWS_S3_CUSTOM_DOMAIN_MEDIA = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME_MEDIA
AWS_LOCATION_MEDIA = 'assets'

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'static')
]

MEDIA_ROOT = os.path.join(BASE_DIR, '/static/media')
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN_MEDIA, AWS_LOCATION_MEDIA)

[codingjoe]

Hi @pdf9,

Thanks for reaching out. At first glance, everything looks correct. Then again, there isn't much to set up to make this package work.

Let's tackle this step by step:

1. Does regular upload work without this package?
2. Does the s3 mock upload work locally? You can check the browsers network logs.
3. Did you check the folder static/tmp? I see that you use the AWS_LOCATION setting. This prefix is also applied to the S3file location.

Let me know if anything unexpected turns up, which examining those points.

Best,
Joe

[pdf9]

Hi Joe,

I tried the s3 mockup and I'm getting bad request notifications when I try to upload - the steps for setup are so simple I feel like I have to be missing something additional that's needed. The only steps I've taken is to install s3file, add to installed apps and middleware in settings, and then add the CORS in s3 (then spent a good long time trying to mess around and get my project to work with it lol). I'm able to upload files through the server to s3 with no issues aside from timeout when I'm running it in heroku. Are there additional things I need to be doing? My project is super simple - have just one project directory and one app directly with a setup like you'd see in the beginner django books - with only models, urls, views, forms, and filter files (I'm still pretty new this).

[codingjoe]

Hi @pdf9,

Thanks for the additional information, but I need a little more. Can you please click on one of the failed requests and post the headers as well as the response body? The response body should contain some indication to the error. If you have a website that you want me to check out for myself (like a Heroku review app, feel free to send a URL to info@johanneshoppe.com).

Best, Joe

[pdf9]

Hi Joe, here are headers and response body of one of the failed requests - thanks so much for your responses/help

Headers:

Response body:

InvalidRequestThe authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.60FC3091CA8B0ABFm6GIJsE647sitsvKZQJCikvkhgZ8SFiJUeOssY8dM3PIa+JgbzurWSNqfQELNJGfx+XMwVt6LUE=

Thanks,
Pat

[pdf9]

After your direction to look at the response body I updated settings to include the below after finding a post on it here: jschneier/django-storages#687

This fixed for me and it's now uploading! I see the tmp folder in my s3 bucket. Thank you so much....seriously you're the man...

if using boto3 (what I am using)
AWS_S3_REGION_NAME = 'us-east-2' #change to your region
AWS_S3_SIGNATURE_VERSION = 's3v4'

if using boto (in case it helps anyone else)
AWS_S3_HOST = 'us-east-2' #change to your region
S3_USE_SIGV4 = True

[pdf9]

Hi Joe,

As stated before the files are now successfully uploading :) but as soon as they finish uploading my browser displays the below json response (looks like that's what should happen based on the view) but FILES are both empty, and the below error appears in my terminal. I think it's telling me there is an issue with actually creating the model records?

In django admin I can see there has been nothing added to the model I created for the test(even though I've uploaded several test files). Is there anything that jumps out to you for why I'm not able to close the loop with uploading to s3 and also creating database records for those files? Or do you think that's even what's happening?

Browser message:

{"POST": {"csrfmiddlewaretoken": "m5wlsyuOxINjqOYG6A0KjImDQmeMjkCiqgquQGuSTztPHScEEejqrtXmcQthC1iH", "progress": "1", "save_continue": "continue_value", "s3file": "["file","other_file"]", "other_file": "[]", "file": "["static/tmp/s3file/BEmMX0_fQAix5E-JTr-N7Q/1.jpg","static/tmp/s3file/BEmMX0_fQAix5E-JTr-N7Q/55.JPG","static/tmp/s3file/BEmMX0_fQAix5E-JTr-N7Q/Capture3.PNG"]"}, "FILES": {"file": [], "other_file": []}}

Django terminal error message:

File not found: statictmp/s3file/69PKsrZ3TsmkobXUwB1P9w/a123.mp4
Traceback (most recent call last):
File "C:\my_project_path\venv\lib\site-packages\storages\backends\s3boto3.py", line 431, in _open
f = S3Boto3StorageFile(name, mode, self)
File "C:\my_project_path\venv\lib\site-packages\storages\backends\s3boto3.py", line 107, in init
self.obj.load()
File "C:\Users\my_project_path\venv\lib\site-packages\boto3\resources\factory.py", line 505, in do_action
response = action(self, *args, **kwargs)
File "C:\Users\my_project_path\venv\lib\site-packages\boto3\resources\action.py", line 83, in call
response = getattr(parent.meta.client, operation_name)(*args, **params)
File "C:\my_project_path\venv\lib\site-packages\botocore\client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\my_project_path\venv\lib\site-packages\botocore\client.py", line 676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\my_project_path\venv\lib\site-packages\s3file\middleware.py", line 34, in get_files_from_storage
f = storage.open(path)
File "C:\my_project_path\venv\lib\site-packages\django\core\files\storage.py", line 36, in open
return self._open(name, mode)
File "C:\my_project_path\venv\lib\site-packages\storages\backends\s3boto3.py", line 434, in _open
raise FileNotFoundError('File does not exist: %s' % name)
FileNotFoundError: File does not exist: static/statictmp/s3file/69PKsrZ3TsmkobXUwB1P9w/a123.mp4

[pdf9]

Hi Joe, ok so I think I have solved the error I was getting but am still not able to create new records for the model. Solved the issue causing the error by just adding a '/' to my AWS_LOCATION so it's 'static/'.

so now something like static/statictmp/s3file/69PKsrZ3TsmkobXUwB1P9w/a123.mp4 is now static/static**/**tmp/s3file/69PKsrZ3TsmkobXUwB1P9w/a123.mp4 and I don't get the error and I see something like this in the json browser response:

{"POST": {"csrfmiddlewaretoken": "MmGTCKMAUWKkIq6DiWk96y24GsJrd2iSQxA20SMEgNqQZukBQADPejDN2WYWwJYh", "progress": "1", "save_continue": "continue_value", "s3file": "["file","other_file"]", "file": "[]", "other_file": "["static/tmp/s3file/kWos2sd4QTC9db8lbO8aMw/4.jpg"]"}, "FILES": {"file": [], "other_file": ["4.jpg"]}}

But I am still not seeing any model records created when I go into django admin after I submit a set of files.

Sorry to keep pestering you with these. I am only sending messages here after I've run up against the wall over and over...thanks again for all your help.

[pdf9]

Hi Joe,

Can you help with this part from the instructions below? I cannot figure out how to save the correct reference to the url in the model instance so that after I run the upload I'm able to reference the urls of the files uploaded using s3file in my templates.

"S3File uploads to a single folder. Files are later moved by Django when they are saved to the upload_to location."

Thank you

Fyi I emailed you the site at info@johanneshoppe.com so you can get an idea of the end goal here.

[codingjoe]

After some investigation I actually did find a bug that is causing your problem. The library currently can't handle custom AWS_LOCATIONs without a trailing slash. I am working on a fix.