Timeout / Continue [Solved]

budfox3 2020-04-11 16:17:04 UTC #1

Hey guys, I'm running a job and went out of runtime. The job (budfox3/projects/thesis/48) was stopped and has currently the status "Timeout". I just purchased a powerup and enabled auto-refresh. Will the job continue to run at some point or do I have to use "rerun"? I just don't want to start from the beginning..

Thanks in advance

redeipirati 2020-04-13 09:08:49 UTC #2

Hi @budfox3,

Will the job continue to run at some point or do I have to use "rerun"? I just don't want to start from the beginning..

You have to rerun it. Did you use any checkpoint strategy during the training? If so, you can mount the output of budfox3/projects/thesis/48 to the next job to resume the training.

Hope that helps.