Getting connection issue when uploading the data

[dabravolski]

Dear community,

First of all, I'm new to the floyd and trying to train my first deep network. For some reason I'm getting the connection issue below when trying to upload my data (the same worked for me yesterday):

(data) Vadzims-MacBook-Pro:digit_recognition vadim$ floyd data upload
Creating data source. Total upload size: 46.1MiB
Uploading files ...
Cannot connect to the Floyd server. Check your internet connection.

Interestingly enough, login/logout works just fine:

(data) Vadzims-MacBook-Pro:digit_recognition vadim$ floyd logout
(data) Vadzims-MacBook-Pro:digit_recognition vadim$ floyd login
Authentication token page will now open in your browser. Continue? [Y/n]: y
Please copy and paste the token from the welcome page.
This is an invisible field. Paste token and press ENTER:
Login Successful
(data) Vadzims-MacBook-Pro:digit_recognition vadim$

I tried to use several network connection (public wifi, home wifi, 4g) without any luck.
Appreciate your help.

[sai]

Oops, that was a bug on our side. We've fixed it and will be releasing an updated version of the CLI later today. Can you please upgrade (pip install -U floyd-cli) and try again tomorrow?

[dabravolski]

Hi, the original issue seems to be address. Thanks for fixing it.

A side question. I have roughly 40k of 64x64x3 images and I'm trying to upload them to floyd. Is it a good idea in general? I'm asking because I can see following messages for about an hour:

(data) Vadzims-MacBook-Pro:digit_recognition vadim$ floyd data upload
Compressing data ...

[dabravolski]

I tried to significantly reduce the data and have just 20 images for upload - and still no luck. I guess there is something off with data upload.

[dabravolski]

@sai, please find the stack trace I see during upload. Let me know if there is something on my side

(data) Vadzims-MacBook-Pro:digit_recognition vadim$ floyd data upload
Compressing data ...
Creating data source. Total upload size: 4.8GiB
Uploading compressed data ...
Traceback (most recent call last): 5198616604/5198616604 - 00:00:00
  File "//anaconda/envs/data/bin/floyd", line 11, in <module>
    sys.exit(cli())
  File "/anaconda/envs/data/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/anaconda/envs/data/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/anaconda/envs/data/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/anaconda/envs/data/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/anaconda/envs/data/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/anaconda/envs/data/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/anaconda/envs/data/lib/python2.7/site-packages/floyd/cli/data.py", line 58, in upload
    data_id = DataClient().create(data)
  File "/anaconda/envs/data/lib/python2.7/site-packages/floyd/client/data.py", line 69, in create
    timeout=3600)
  File "/anaconda/envs/data/lib/python2.7/site-packages/floyd/client/base.py", line 57, in request
    self.check_response_status(response)
  File "/anaconda/envs/data/lib/python2.7/site-packages/floyd/client/base.py", line 81, in check_response_status
    response.raise_for_status()
  File "/anaconda/envs/data/lib/python2.7/site-packages/requests/models.py", line 909, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://www.floydhub.com/api/v1/modules/

[naren]

@dabravolski Hey! Seems like the dataset is too large causing the timeout. We have tested data up to 2GB. Can you try uploading a subset of your dataset?

Alternatively, you can also download data directly into Floyd if it is available on the internet. See this doc for instructions on this:

http://docs.floydhub.com/home/using_datasets/#creating-a-dataset-from-downloads

[paulantoine]

hello,
same issue, same error message (but with a 1.1 GB / 800 Mo compressed data)

[alutrin]

Hi @naren, you need to test with larger data. 2GB is tiny in ML. I'm failing uploading 7GB ( File "/Users/alutrin/anaconda2/lib/python2.7/site-packages/requests/adapters.py", line 499, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.floydhub.com', port=443): Read timed out. (read timeout=3600))

If 2GB is the limit then we can't use floydhub for any serious ML datasets. I will try to download from a script but the drawback is that it will cost money

Update: downloading from a script works well, but I would still prefer to use the upload functionality even for large datasets.

[mckay]

To those still following this thread, this issue should have been resolved in floyd-cli version 0.9.8. There is no longer a size limit to data uploads.

If anybody still sees problems, please post them here.

[jnsaulo]

Hi, I am having a trouble with uploading my data set to the cloud. My data set contains lots of pictures which has a size of 36.4 GB. I already did upgrade my floydhub in my laptop. I am using an ubuntu os. I will really appreciate if you reply as soon as possible. Thank you,

[lusob]

I'm getting the same problem when I'm trying to upload an 8Gb images dataset, when I tried to resume the upload it said it's corrupted, try to upload a new one. I repeat the process several times without success. Any help would be greatly appreciated

[mnalavadi]

Will the data limit be increased anytime soon? I was having issues with 1.6GB..

[redeipirati]

Hi @mnalavadi,

We are working on dataset acquisition to improve it. Are you experiencing an infinite pending state? Are you uploading a dataset taken directly from Internet or collected by you? If you can provide these infos, I can provide you more help.

[mnalavadi]

Hi, thanks for the quick reply. I was experiencing an infinite pending state, which I deleted this morning and am currently reuploading. I made the dataset myself. I got this error on my command line:

TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

But as others have mentioned, was able to connect to login or and floyd status.

Thanks

[redeipirati]

Thank you for the infos @mnalavadi, I have reported this to the dev team, we will update you as soon as possible.

[mnalavadi]

Thanks. The re-upload looks to have worked, but I'm still not sure why it didn't work the first time around, the data was exactly the same both times.

[omarsalimwaller]

I'm having this same issue with the most up to date cli.

[houqp]

Hi @omarsalimwaller, could you let me know your floydhub username so I can follow up with you privately? I would need to you run a special command to get me more debug info.