[Solved] Tensorboard memory leak

dldx 2018-09-10 12:51:15 UTC #1

Hi,

Tensorboard seems to have a bad time if there are lots of files in the home folder. As part of managing a new dataset, I had to create around 200,000 image files. Tensorboard nearly filled up all the memory and crashed my session.

I suggest making Tensorboard point to a logs folder rather than the home folder in general. What's worse is that if you try to kill tensorboard, it comes back.

Cheers,
Durand

redeipirati 2018-09-11 08:22:03 UTC #2

Hi Durand,

Thanks for reporting this, the Dev Team is currently investigating the memory leak issue, we will update you as soon as possible.

houqp 2018-09-11 17:16:10 UTC #3

Hi @dldx,

We have been working on a solution to address this problem for the last couple days. I am hoping to get the code change out by end of this week. During the last couple weeks, we have noticed many memory leak issues from tensorboard, it not only leaks memory with large directories, but also with directory that has a single large tf event file. So pointing it to logs folder won't fix all the leaks.

After the our tensorboard bandage is out, I will see if I can help fix those actual memory bugs in tensorboard and contribute the patches back to upstream.

dldx 2018-09-12 15:33:19 UTC #4

Glad to hear it! I guess tensorboard hasn't been deployed as extensively before as on FloydHub.

houqp 2018-09-13 21:15:05 UTC #5

@dldx, we were able to track down the memory leak issue, it's due to an inefficient implementation of directory walk function from tensorflow. It's going to take some amount of time for us to create a fix and have it roll out to all platforms. While we are working on it, I have pushed out a temporary fix to prevent tensorboard process from consuming memory over a given treshold, this should stop it from crashing the machine. Could you give it another try?