Select data source when creating a dataset (e.g. AWS S3 bucket or Google Cloud bucket)

[jackvial]

I started using Floyd Hub a few days ago and I'm really loving the convenience compared to spinning up EC2 instances.
But one feature I think would improve the platform is being able to access AWS s3 bucket data and Google Cloud buckets. This would be great for very large datasets that might otherwise take days to download and upload on a slow connection.

This is how I imagine it would work from a users perspective:
1. Create new dataset
2. Select dataset source (options might be s3, Google Cloud Bucket, etc)
3. Enter bucket URL and credentials. (This might create a link from the dataset to the bucket or copy it)
4. Read data from bucket in Python, Jupyter etc..

[sai]

Thanks for the feature request. This is high on our priority and we'll be building integrations with S3 in the very near future. Will keep you posted!

[jackvial]

Excellent! Looking forward to it

[craigboman]

Any update on this feature request? Looking forward to this option as well.

[redeipirati]

Hi @craigboman,

Any update on this feature request? Looking forward to this option as well.

it's currently on our roadmap, but it could take a while. Meanwhile, here's what you can do if you are working with S3 bucket.

Our enterprise users connect to their private S3 buckets from within their FloydHub job. You can create S3 credentials with read-only access to specific buckets and then use the awscli tool to download the data. You can either pull the data once and create a dataset out of it. Then you can mount the data at runtime to your training job. This is the approach we recommend to customer with large datasets in S3.

Alternatively, you can also stream the S3 data every time the training starts - this would require some changes to the training code. And it is a little slow to stream S3 data across the internet for every training job.

Hope that helps.