aws s3 ls - find files by modified date?

[ChrisSLT]

Hi,
We'd like to be able to search a bucket with many thousands (likely growing to hundreds of thousands) of objects and folders/prefixes to find objects that were recently added or updated. Executing aws s3 ls on the entire bucket several times a day and then sorting through the list seems inefficient. Is there a way to simply request a list of objects with a modified time <, >, = a certain timestamp?

Also, are we charged once for the aws s3 ls request, or once for each of the objects returned by the request?

New to github, wish I knew enough to contribute actual code...appreciate the help.

[jamesls]

The S3 API does not support this, so the only way to do this just using S3 is to do client side sorting.

As far as S3 pricing, we use a ListObjects request which returns 1000 objects at a time. So you will be charged for a LIST request per every 1000 objects when using aws s3 ls.

Another alternative is to store an auxiliary index outside of S3, e.g dynamodb. Let me know if you have any other questions.

[ChrisSLT]

Thank you

[jwieder]

Although this functionality appears to remain absent from aws-cli, its pretty easy to script it in bash. For example:

#!/bin/bash
DATE=$(date +%Y-%m-%d)
aws s3 ls s3://bucket.example.com/somefolder/ | grep ${DATE}

[PuchatekwSzortach]

@jwieder This doesn't help user decrease number of list calls to s3. Say that every day you store ~1000 news articles in a bucket. Then on client side want to get articles for last 3 days by default (and more only if explicitly requested). Having to fetch a list of all the articles since the beginning of time, say 100k, takes time and accrues network costs (because a single list call will return only up to 1000 items). It would be much nicer to be able to say "Give me a list of items created/modified since 3 days ago".

[ChrisSLT]

Exactly!

On Sun, Jan 17, 2016 at 11:53 PM, PuchatekwSzortach <
notifications@github.com> wrote:

@jwieder https://github.com/jwieder This doesn't help user decrease
number of list calls to s3. Say that every day you store ~1000 news
articles in a bucket. Then on client side want to get articles for last 3
days by default (and more only if explicitly requested). Having to fetch a
list of all the articles since the beginning of time, say 100k, takes time
and accrues network costs (because a single list call will return only up
to 1000 items). It would be much nicer to be able to say "Give me a list of
items created/modified since 3 days ago".

—
Reply to this email directly or view it on GitHub
#1104 (comment).

[jwieder]

@PuchatekwSzortach @ChrisSLT You're right, sorry for my lame reply; and I agree this sort of functionality would be very helpful in aws-cli. The combination of leaving this basic feature out and billing for file listings is highly suspect. Until AWS stops penny-pinching and introduces listing by file properties, here's another idea that I've used that is more relevant to this thread then my 1st reply: For files that need to be tracked in this way, files are named with a timestamp. A list of files is stored in a local text file (or could be db if you have gazillions of files to worry about). Searching for a date then involves opening the file, looking for filenames that match the today's date could look something like this:

while read -r fileName
do
if [ "$fileName" == "$TODAY" ]; then
aws s3 sync $BUCKETURL /some/local/directory --exclude "*" --include "$fileName"
fi
done < "$FILE"

Where $FILE is your local filename index and $TODAY is the date you are searching for. You'll need to change the condition on this loop, but hopefully this can give you an idea.

Doing things this way relieves you of any charges related to listing the files in your bucket; but it also depends on the client you are conducting the search on having access to the local file list ... depending on your application / system architecture that might make this sort of approach unfeasible. Anyway, hope this helps and apologies again for my earlier derpy reply.

[ChrisSLT]

Agreed and thank you

On Tue, Jan 19, 2016 at 10:00 AM, Josh Wieder notifications@github.com
wrote:

@PuchatekwSzortach https://github.com/PuchatekwSzortach @ChrisSLT
https://github.com/ChrisSLT You're right, sorry for my lame reply; and
I agree this sort of functionality would be very helpful in aws-cli. The
combination of leaving this basic feature out and billing for file listings
is highly suspect. Until AWS stops penny-pinching and introduces listing by
file properties, here's another idea that I've used that is more relevant
to this thread then my 1st reply: For files that need to be tracked in this
way, files are named with a timestamp. A list of files is stored in a local
text file (or could be db if you have gazillions of files to worry about).
Searching for a date then involves opening the file, looking for filenames
that match the today's date could look something like this:

while read -r fileName
do
if [ "$fileName" == "$TODAY" ]; then
aws s3 sync $BUCKETURL /some/local/directory --exclude "*" --include
"$fileName"
fi
done < "$FILE"

Where $FILE is your local filename index and $TODAY is the date you are
searching for. You'll need to change the condition on this loop, but
hopefully this can give you an idea.

Doing things this way relieves you of any charges related to listing the
files in your bucket; but it also depends on the client you are conducting
the search on having access to the local file list ... depending on your
application / system architecture that might make this sort of approach
unfeasible. Anyway, hope this helps and apologies again for my earlier
derpy reply.

—
Reply to this email directly or view it on GitHub
#1104 (comment).

[willstruebing]

There is a way to do this with the s3api and the --query function. This is tested on OSX
aws s3api list-objects --bucket "bucket-name" --query 'Contents[?LastModified>=2016-05-20][].{Key: Key}'
you can then filter using jq or grep to do processing with the other s3api functions.

Edit: not sure why they are not showing up, but you have to use backticks to surround the date that you are querying

[snandyala]

Is it possible for you to create folders for each day and that way, you will be accessing only todays files or at most yesterdays folders to get the latest files.

[willstruebing]

yes. Although you may find it easier to simply use a date prefix for your keys (you cannot query a bucketname/foldername combination using the --bucket option). Using the date prefix will allow you to use the --prefix flag in the cli and speed up your queries as AWS recommends using numbers or hashes at the beginning of key names for increased response times.

[kislyuk]

@willstruebing, your solution still does not reduce the number of S3 API calls, server-side query complexity, or amount of data sent over the wire. The --query parameter performs client-side jmespath filtering only.

[willstruebing]

@kislyuk I agree completely that is does not answer the efficiency issues. However, my intention was to answer the specific question:

Is there a way to simply request a list of objects with a modified time <, >, = a certain timestamp?

That basic question is how I ended up on this thread, and so I thought it reasonable to include an answer to it. The issue is labeled "aws s3 ls - find files by modified date?".

I would love to hear anyone's ideas on the efficiency parts of the question, as I don't have one myself and am still curious.

[sreeninair]

#for i in s3cmd ls | awk {'print $3'} ; do aws s3 ls $i --recursive ; done >> s3-full.out