read_csv Pass File System (AWS, GCP) as Option.

[phdnzdeveloper]

Code Sample

pd.read_csv('s3://other_owner_bucket_name/data', file_system=s3fs.FileSystem(key='abc', secret='123'))

Or

pd.read_csv('s3://other_owner_bucket_name/data', storage_options={'key':'abc', 'secret':'123'})

Problem description

Hi Team, I am currently using pandas for query a lot of CSV data from different file system such as AWS, GCP. However, with a current option, it just allows default key/secret credential from configuration file [default]. So I would like to ask for an option to pass a pre-created file system to read CSV files from different clouds with different credentials.

Best Regards
Giang

[TomAugspurger]

Seems reasonable. Are you interested in working on this?

The first step would be to write a detailed docstring so that we can understand how things work and interact with the other parameters in read_csv.

Would this also apply to to_csv? What about other readers?

[jreback]

note that was a prior PR that was stalled to pass credential in (closed now) might be a good starting point -

[suzutomato]

Let me try and work on this, unless @phdnzdeveloper or someone else are willing to.
I'll read s3fs and gcsfs docs as well as the prior PR, then start with a doc string.

[phdnzdeveloper]

Hi Everyone,

Sorry! I have just come back from holiday. I have posted a doc string for explanation.

Regards
Giang

storage_options : dict, optional
A dictionary of parameters to pass on as part of filepath_or_buffer URL including s3://, gcs://, ... which based on different file systems such as s3fs, gcsfs, ...
example: pd.read_csv('s3://bucket_name/key_name.csv', storage_options={'key':'abc', 'secret':'123'}).

[suzutomato]

Thank you! Then I'll release this (by removing my comment "take", hopefully this works as intended)

[JMBurley]

Weirdly, s3fs documentation acts like the storage_options are already live in pandas, even though this is only due to go live in 1.2 https://s3fs.readthedocs.io/en/latest/ .

Which isn't a pandas problem, but I do want to mention it here so anyone else who goes on a google hunt gets an answer to why that is the case somewhat faster than I did...

[TomAugspurger]

cc @martindurant about the s3fs docs. Pandas 1.2 probably won't be released till November / December.

This issue was closed by #35381 though.