Queue jobs when exceeds maximum concurrent jobs

Coxious 2018-02-18 15:05:27 UTC #1

Hi,
Sometimes I want to submit more than 20 jobs, which exceeds the limitation of the concurrent jobs, can we add an option to the floyd-cli and queue the jobs into a queue, and let them run in order?

Best,
Coxious

houqp 2018-02-18 19:25:21 UTC #2

Sure! I am working on this now, will try to get it out next week.

Coxious 2018-03-19 14:31:35 UTC #3

Hi @houqp,

I've been using this functionality, very convenient! Can you add a button to cancel all queued jobs with a single click, and maybe add a group option so I can make some of the jobs belong to a specified group and fold them. Also, job queue with maximum capacity 20 is still a bit small, can I expend them to 50? Thanks very much!

Best,
Coxious

houqp 2018-03-20 22:10:03 UTC #4

@Coxious, you are in the datascientist plan, so you should already have a queue size of 50. Where were you seeing the 20 limit? Beginner plan as the queue size limit set to 32.

We are planning on adding more job management features to the next version of the job list UI that's under design.

Coxious 2018-03-22 10:36:19 UTC #5

Hi @houqp,

I saw that limit when I trying to submit a new job using floyd run command. I was using GPU instances. Including those jobs running, I can submit 20 jobs maximum. I just tried to get the error information again but it looks like CPU instances support more than 20 jobs. I don't have that much of the actual training jobs at the moment to test the GPU queuing, but I am pretty sure since I just saw the error 2 days ago. Can you check with that?

Best,
Coxious

houqp 2018-03-22 23:22:38 UTC #6

Ha, you are right, that's a very old restriction we put in to protect our users a long time ago! We will remove that restriction this week. It only applies to GPU instances. CPU, CPU2 and GPU2 are not affected. Thanks for the report!

houqp 2018-03-27 00:52:35 UTC #7

@Coxious, 20 concurrent job restriction has been lifted.