OSError: Failed to interpret file %s as a pickle [Solved]

[tirth]

I have a dataset of 20Gb's which is of Numpy array having the extension ".npy". My single file of .npy is of 20Gb's

When I try to upload that 20Gb file using the command "floyd data upload" it compressed the 20Gb's of data into only 19.8Mb's which is not possible because you can't compress 20Gb's file to 19.8Mb's. Seeing at this compression I was shocked and try to found out how to compression works in floyd-cli but couldn't.

Terminal:

But when I tried to load the array using

X = np.load('/floyd/input/data/crop_256_trainLabels_master_256_v2.npy')

Then I somewhat get the idea of how compression work in floyd-cli. I referred the StackOverflow link for the link which shows the same issue https://stackoverflow.com/questions/19793937/failed-to-interpret-file-s-as-a-pickle-when-loading-an-npy-array

The solution on StackOverflow says that if you save the ".npy" array using the mention below method then it is not the perfect way of saving the ".npy" array
Method:

f = open(Filename, "w")
try:
f.write(a)

I was not sure if I'm right or wrong about the compression, but the way compression works for ".npy" array is not proper and if you want you can compress the ".npy" array using the Numpy compression method which converts the ".npy" into ".npz"

Please help! Because I have to submit my project and the deadlines are near. Thank You!

[tirth]

It shows the OSError

[tirth]

Original Data:

[redeipirati]

Hi @tirth,

When I try to upload that 20Gb file using the command "floyd data upload" it compressed the 20Gb's of data into only 19.8Mb's which is not possible because you can't compress 20Gb's file to 19.8Mb's.

In reality, it's possible to compress 20Gb in even less than 20Mb, but this is not certainly the case.

Seeing at this compression I was shocked and try to found out how to compression works in floyd-cli but couldn't.

On each floyd data upload, the CLI will create a gzip compressed archive of the current folder. Here are the lines of code.

I'm not sure what is happening here. Is this file crop_256_trainLabels_master_256_v2.npy publicly available from the internet?

[tirth]

The dataset is locally created by me and it was not publicly available.

I have found out that. The compression only compress this crop_256_trainLabels_master_256_v2.npy File into 19.8Mbs. But when i try to compress the another file which of same 20Gbs it was compressed to 9.9Gbs which if usually the case for “.npy“

Because if i try to compress locally with the Numpy then is usually output file around 9.9Gbs having extension “.npz”.

So is there’s some problem in crop_256_trainLabels_master_256_v2.npy? I’ll try to recreate the file and then again try to upload it.

Is there’s any another option to upload the large file using Browser though drag and drop?

[tirth]

Hi @redeipirati
Here is the dataset link:- https://www.floydhub.com/tirth/datasets/cooldata/

If you try to load this array using

x = np.load("crop_256_trainLabels_master_256_v2.npy")
print(x.shape)

Then it shows the same error.

[redeipirati]

Hey @tirth,

The link you provide is no longer available. Can you provide it again? Did you check if there was any error in the data generation script? Other question: during the data load on FloydHub did you use the same python version used on the local data preparation?

[tirth]

I deleted the link. But when is tried to upload the 20Gbs with the computer having 32Gbs of RAM then it perfectly compress it. Thanks for you support