FUSE integration to mount large dataset

green405 2019-10-21 18:24:42 UTC #1

I am trying to install fuse-zip package to mount a zip file and work with the files in it without extracting any of those files. Since I'am unable to unzip large files due to space constraints in the Floydhub environment. I am trying to use fuse-zip as a work around. When I follow the steps below, I am unable to access the package that is installed.

#Step One
!apt-get install -y fuse-zip

#Step Two
# 30-45min in Google Colab.
raw_data_dir = "input/raw"
!kaggle competitions download -c rsna-intracranial-hemorrhage-detection -p {raw_data_dir}

#Step Three
# Check that everything is working.
!ls {input_dir}

When I run step two, I get package unfound error:

E: Unable to locate package fuse-zip

redeipirati 2019-10-22 08:06:36 UTC #2

Hi @green405,

The cache of aptitude is empty (we do this to reduce the environment size), here's the fix:

# Step One
!apt-get update && apt-get install -y fuse-zip

...

Hope that helps

green405 2019-10-22 15:05:36 UTC #3

Hi @redeipirati,

Thank you very much, this helps out a lot. However, I encounter another error once everything is installed.

%%time
# Around 10 min in Google Colab.

input_dir = "input/raw/kaggle_data"
!mkdir {input_dir}
!fuse-zip input/raw/rsna-intracranial-hemorrhage-detection.zip  {input_dir}

#error message
fuse: device not found, try 'modprobe fuse' first
CPU times: user 10.1 s, sys: 3.55 s, total: 13.6 s
Wall time: 9min 4s

I don't have a clue on how to address the modprobe fuse issue. I googled search and came up empty or not as technical to figure that piece out.

redeipirati 2019-10-23 08:24:34 UTC #4

Hi @green405,

Sorry, I got it now what you are trying to do. Unfortunately we don't support FUSE at this time because supporting this feature might cause vulnerabilities issue on our environments. We are still experimenting with different ways to improve the experience with large dataset, but it could take a while.

I have updated the title / category to reflect that this is a new feature request. This will help us gauge interest from our users.

Thanks again for the Feature Request.

green405 2019-10-23 12:29:01 UTC #5

Hi @redeipirati,

Thank you very much for your help. Do you of any alternative to dealing with large zip files with images? Is there a way just to read the zipfile without extracting the files?

redeipirati 2019-10-24 09:08:10 UTC #6

Do you of any alternative to dealing with large zip files with images? Is there a way just to read the zipfile without extracting the files?

Unfortunately, I do not have any alternative in mind at the moment. I'll keep you posted if I'll find some workaround.