How to run Tensorflow Object Detection API

[ashwin4ever]

I am using the Tensorflow Object Detection API from here Object Detection Models. Based on this tutorial Tensorflow tutorial, I am able to get it running in my local system. However being very slow I decided to try it out on FloydHubs GPU servers. When I issued the run command from the object detection folder, I got the Code size too large restraint. I then uploaded the data(which is essentially the labels and the tfrecords for the images) as a separate folder. Despite this I need a config file which is used by tensorflow to pick various parameters and in that I need to specify the checkpoint file which is by default 104MB. If I upload this separately, how do I specify the path on the server in my config file. How do I also specify the data path in the server inside my config file since I only need to run the train.py file using this cmd locally
python train.py --logtostderr --train_dir=training\ --pipeline_config_path=training\sssd_mobilenet_v1_pets.config

These are relevant lines in the config file:
fine_tune_checkpoint: "ssd_mobilenet_v1_coco_11_06_2017/model.ckpt"
eval_input_reader: {
tf_record_input_reader {
input_path: "data/test.record"
}
label_map_path: "data/flower_detect.pbtxt"
shuffle: false
num_readers: 1
}

This is my first time and I would really appreciate a guiding hand since the environment is new to me

Update:
Changed the paths to: https://www.floydhub.com/ashwin4ever/datasets/ssd_mobilenet_v1_coco_11_06_2017/1/training/model.ckpt. Not sure if this is going to work.

I used the command line version of python train.py but get an error of the nature ImportError: No module named 'object_detection' For this to work, the following command must be executed export PYTHONPATH=$PYTHONPATH:pwd:pwd/slim. I am not sure how to do that in Floydhub. Would really appreciate any response

[redeipirati]

Hi @ashwin4ever,

First of all, you need to mount the dataset(data and model) in this way: (I'll use your last run as reference)

floyd run --data ashwin4ever/datasets/ssd_mobilenet_v1_coco_11_06_2017/1:model --data ashwin4ever/datasets/data/1:dataset --gpu --mode jupyter

In this way, you can:
- access ssd_mobilenet_v1_coco_11_06_2017 from the /model folder
- access data from the /dataset folder

Then inside the config file:

fine_tune_checkpoint: "/model/ssd_mobilenet_v1_coco_11_06_2017/model.ckpt"
eval_input_reader: {
tf_record_input_reader {
input_path: "/dataset/data/test.record"
}
label_map_path: "/dataset/data/flower_detect.pbtxt"
shuffle: false
num_readers: 1
}

Inside the Terminal, type export PYTHONPATH=$PYTHONPATH:$(pwd):$(pwd)/slim make sure to be in the directory with the slim package before run this command.

Hope it helps.

[dirkpitt]

I'm also going through the usage of the same API, and there are a few issues dealing with floydhub infrastructure that I couldn't solve
1. To link the pipeline.config file I have to mount it as dataset, but this is very inconvenient because it should be part of the code versioning. It is feasible tough.
2. According to the API documentation, I have to run the train.py and the eval.py jobs at the same time, mounting the output of the former as input for the latter. However it seems like the output of the train.py job is not available for mounting until the job is not finished, which makes the overall logic fail. Am I wrong, or is there a way to make it work?

[redeipirati]

Hi @dirkpitt,

To link the pipeline.config file I have to mount it as dataset, but this is very inconvenient because it should be part of the code versioning. It is feasible tough

Why do you need to mount this config file as a dataset? Does the size of this file take more than 150MB?

According to the API documentation, I have to run the train.py and the eval.py jobs at the same time, mounting the output of the former as input for the latter. However it seems like the output of the train.py job is not available for mounting until the job is not finished, which makes the overall logic fail. Am I wrong, or is there a way to make it work?

I have 2 possible solutions in mind:

1. You can create a wrapper (a bash or python script) which sequentially runs the train and evaluate scripts. Unfortunately, the scripts don't use the tf.Estimator API, otherwise it would be simple to create a train_and_eval.py script.

2. You can launch an interactive Job (--mode jupyter), open a Terminal e sequentially launch the train and eval scripts manually.

[dirkpitt]

Regarding 1. I have to pass the path to this file as command line argument, i.e. "python train.py --pipeline=/pipeline.config", so I need the path to the pipeline.config file. I couldn't find in the docs what's the path of files in the same folder as the python code that's been run.

Regarding 2. If I understood rightly tensorflow API, the two jobs must run at the same time, and not sequentially.

[redeipirati]

Regarding 1. I have to pass the path to this file as command line argument, i.e. "python train.py --pipeline=/pipeline.config", so I need the path to the pipeline.config file. I couldn't find in the docs what's the path of files in the same folder as the python code that's been run.

If you sync/upload the config file with your code, you only need to reference it from the local folder as a relative path. Here's some example:

• case where pipeline.config file is in the same folder of train.py:
  python train.py --pipeline=pipeline.config

• case where pipeline.config file is in a local subfolder called config:
  python train.py --pipeline=config/pipeline.config

Regarding 2. If I understood rightly tensorflow API, the two jobs must run at the same time, and not sequentially.

The default behavior is exactly what you have reported, but the docs seems a bit outdated since there is a step parameter. You can also see the parameter in the config sample. In this way, you can run sequentially.

[dirkpitt]

Actually I am already using the num_step parameters, and I set it to 100 - very low, just to try. And yet, when I run the eval.py after the train.py has finished, the job is stuck, a bit like described in this post https://github.com/tensorflow/models/issues/2225 (or this one https://stackoverflow.com/questions/47713286/tensorflow-object-detection-api-evaluation-stuck)
The output of my job is

2018-02-19 01:04:49,372 INFO - from ._conv import register_converters as _register_converters
2018-02-19 01:04:53,361 INFO - INFO:tensorflow:depth of additional conv before box predictor: 0
2018-02-19 01:04:53,391 INFO - INFO:tensorflow:depth of additional conv before box predictor: 0
2018-02-19 01:04:53,422 INFO - INFO:tensorflow:depth of additional conv before box predictor: 0
2018-02-19 01:04:53,452 INFO - INFO:tensorflow:depth of additional conv before box predictor: 0
2018-02-19 01:04:53,482 INFO - INFO:tensorflow:depth of additional conv before box predictor: 0
2018-02-19 01:04:53,512 INFO - INFO:tensorflow:depth of additional conv before box predictor: 0

and it hangs there...

[redeipirati]

Have you tried to set the max_eval parameter as suggested by this user?

By default, the eval script runs forever. You'll need to define max_evals in your eval proto in your train config proto.

[dirkpitt]

Yes, it's all set. Here's the relevant part
eval_config: {
num_examples: 20
max_evals: 20
num_visualizations: 10
eval_interval_secs: 60
}

I am wondering whether you (floydhub staff) have ever tried to run the object detection api in floydhub? There are several pitfalls so it would be a nice help.

At the moment I am stucked with this, and if I cannot run the eval job then I am not confident in using the results of the train.py job...

[redeipirati]

You are right, this project has a couple of pitfalls at running on FloydHub because it was born for distributed/multi-gpu settings in mind. During the next days, I'll try to port it to FloydHub.

[valiok98]

Hi, I am following the same tutorial and am currently having trouble exporting the trained model. After I interrupt the training process , there is nothing in the /output directory. I currently specify nowhere in the code, wheter to use /output directory, but as stated in the tutorial one should expect the checkpoints in the training directory. Any help would be greatly appreciated . Please ..
All the best

[redeipirati]

Hi @valiok98,

I don't think that this works

saver.save_path = '/output/'

A simple workaround is to save the content of the training dir after the training script, in this way:

floyd run ...
'python train.py --logtostderr --train_dir=output/ --pipeline_config_path=output/ssd_mobilenet_v1_pets.config && cp output/* /output/'

Hope it helps.

[valiok98]

cp output/* /output/'

Hi man, unfortunately it still does not work for me. Since you've got intel on my commits can you check my latest one and tell me what I am doing wrong. I have renamed the folder again to just training.

EDIT : never mind I figured it out and for others with the same problem: Simply reduce the training steps , so that your model terminates by itself. If you cancel the training it happens to not save the model checkpoints in the output directory. Thanks again ! Great community here.

[AshiqKS]

@redeipirati, @valiok98, I am trying the object detection API with the following command 'floyd run --env tensorflow-1.5 --data https://www.floydhub.com/ashiqks/datasets/dimg/1:license "python train.py --logtostderr --training_path=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config" '.
The data in the dimg is again two subfolders, train and test.
For both the training_path and pipeline_config_path I used the local folders in my system.
There is an error showing 'ValueError: too many values to unpack <expected 2 '
Could anyone please help me to solve this problem.

[redeipirati]

Hi @AshiqKS,

There is an error showing 'ValueError: too many values to unpack <expected 2 '

Can you provide the link to the FloydHub's Job with the error, please? If not, even the full Log is fine. This will help us with the issue.

You can also take a look at the last @valiok98 Job about how to structure the code/execution: https://www.floydhub.com/valiok98/projects/image_raspberrypi2/82

[AshiqKS]

Dear @redeipirati, not even a job was created with the command line. I changed the commanline to 'floyd run --gpu --env tensorflow-1.5 --data ashiqks/datasets/dimg/1:/data 'python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config && cp training/* /output/' as referring to the job link you gave me, there was new error 'Error : no such option : --logtostder
I initialised the project on the 'object_detection' sub-folder on the 'research' folder of the root folder.

[redeipirati]

--logtostder

Is possible that this parameter was introduced by @valiok98. Try to run without it.

I initialised the project on the 'object_detection' sub-folder on the 'research' folder of the root folder.

Seems fine, this is the right folder: https://github.com/tensorflow/models/tree/master/research/object_detection

[AshiqKS]

@redeipirati, --logtostderr is a default parameter. Even after eliminating it the error was on --train_dir.

[redeipirati]

This is really odd because this command is correct:

floyd run --gpu --env tensorflow-1.5 --data ashiqks/datasets/dimg/1:/data 'python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config && cp training/* /output/'

Have you checked the single quote? It seems like a command formatting error...

[valiok98]

@redeipirati @AshiqKS I know the error. You might have copied the command straight from the floyd job window. That is not correct because it is displayed with single quotes ' , and you must replace them with double ones " in order to work for you in your terminal
LOL just saw you have mentioned that in your comment, so yeah