Open in app

Sign in

Write

Sign in

Waymo Open Dataset: Tensorflow 2 Object Detection API

許嘉倫
5 min readAug 13, 2021

--

GitHub Repository

gyes00205/waymo_tf_object_detection

Download Dataset

download link: Waymo Open Dataset
domain_adaptation directory doesn’t have label data, so please download the data in training directory

tfrecord details

  • 5 kinds of camera photos and Lidar informations
  • num_classes: 0 — Unknown, 1 — Vehicle, 2 — Pedestrian, 3 — Sign, 4 — Cyclist
    In this project, we don’t need sign and Unknown classes, so we should modify label_map.pbtxt :
item {
    id: 1
    name: 'vehicle'
}item {
    id: 2
    name: 'pedestrian'
}item {
    id: 4
    name: 'cyclist'
}
  • camera categories: FRONT, FRONT_LEFT, FRONT_RIGHT, SIDE_LEFT, SIDE_RIGHT
  • bbox (x, y, w, h) coordinate: (x, y) represents center coordinate of bbox, (w, h) represents width and height.

Setup Environment

Install Waymo open dataset

pip3 install waymo-open-dataset-tf-2-1-0==1.2.0

Install COCO API

pip install cython
pip install git+https://github.com/philferriere/cocoapi.git#subdirectory=PythonAPI

Install Tensorflow 2 Object Detection API

Refer to TensorFlow 2 Object Detection API tutorial to install toolkit

Step1: git clone Tensorflow 2 Object Detection API

git clone https://github.com/tensorflow/models.git

Step2: go to models/research/ and execute

protoc object_detection/protos/*.proto --python_out=.

Step3: add API to your environment path

export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

Step4: copy setup.py to models/research/

cp object_detection/packages/tf2/setup.py ./

Step5: install setup.py

python -m pip install .

Step6: Test whether the installation is successful

python object_detection/builders/model_builder_tf2_test.py

Step7: build directory structure

Waymo
├───models #Tensorflow Object Detection API
├───training_configs #training config
├───pre-trained-models #pretrained model
├───exported-models #exported model
└───data #training data
    └───segment-???.tfrecord

Convert tfrecord format

Besides of Lidar informations in waymo’s tfrecord, the below is its bbox format:

(x0, y0): is center coordinate. (w, h): is width and height.

(x0, y0): 為中心點座標。 (w, h): 為長寬。

Our goal is to filter out Lidar and convert bbox to the following format:

(x1, y1): is left-top coordinate. (x2, y2): is right-down coordinate.

(x1, y1): 為左上角座標。 (x2, y2): 為右下角座標。

The reference code that convert tfrecord is LevinJ/tf_obj_detection_api, and make some minor changes.

create_record.py:

  • filepath: the path of tfrecord
  • data_dir: the converted tfrecord will be stored in the data_dir/processed directory

Execute code:

python create_record.py \
--filepath=data/segment-???.tfrecord \
--data_dir=data/

After executing th code, the processed tfrecord will appear in data/processed directory.

Waymo
├───models
├───training_configs 
├───pre-trained-models 
├───exported-models 
└───data
    ├───processed
    │   └───segment-???.tfrecord # processed tfrecord
    └───segment-???.tfrecord

Download pretrained model

go to Tensorflow Model Zoo and download pretrained model.

I download SSD ResNet50 V1 FPN 640x640 (RetinaNet50) pretrained model.

Step1: go to pre-trained-models directory.

cd pre-trained-models

Step2: download SSD ResNet50 pretrained model

wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz

Step3: unzip the file

tar zxvf ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz

Waymo
├───models
├───training_configs 
├───pre-trained-models 
│   └───ssd_resnet50_v1_fpn_640x640_coco17_tpu-8
│       ├─ checkpoint/
│       ├─ saved_model/
│       └─ pipeline.config
├───exported-models 
└───data
    ├───processed
    │   └───segment-???.tfrecord #processed tfrecord
    └───segment-???.tfrecord

Modify training config

Go to configs/tf2, and find corresponding config that is ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.config

Step1: create folder in training_configs directory

cd training_configs
mkdir ssd_resnet50_v1_fpn_640x640_coco17_tpu-8

Step2: create pipeline.config

Create pipeline.config in ssd_resnet50_v1_fpn_640x640_coco17_tpu-8 directory. Copy and paste the config content you just found, and make some modifications.

  • num_classes: number of classes
  • batch_size: according to your computer memory
  • fine_tune_checkpoint: modify to pretrained model ckpt-0 path
  • num_steps: training steps
  • use_bfloat16: whether to use tpu, if not used, set to false
  • label_map_path: label_map.pbtxt path
  • train_input_reader: set input_path to the tfrecord path for training
  • metrics_set: “coco_detection_metrics”
  • use_moving_averages: false
  • eval_input_reader: Set input_path to the tfrecord path for evaluating
Waymo
├───models
├───training_configs 
│   └───ssd_resnet50_v1_fpn_640x640_coco17_tpu-8
│       └───pipeline.config # create pipeline.config
├───pre-trained-models 
│   └───ssd_resnet50_v1_fpn_640x640_coco17_tpu-8
│       ├─ checkpoint/
│       ├─ saved_model/
│       └─ pipeline.config
├───exported-models 
└───data
    ├───processed
    │   └───segment-???.tfrecord
    └───segment-???.tfrecord

Train Model

model_main_tf2.py

  • model_dir: the training checkpoint will be stored in the model_dir directory
  • pipeline_config_path: pipeline.config path

Execute code:

python model_main_tf2.py \
--model_dir=training_configs/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8 \
--pipeline_config_path=training_configs/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/pipeline.config

Execution results: it will be printed every 100 steps.

Step 2100 per-step time 0.320s
INFO:tensorflow:{'Loss/classification_loss': 0.121629156,
 'Loss/localization_loss': 0.16370133,
 'Loss/regularization_loss': 0.2080817,
 'Loss/total_loss': 0.4934122,
 'learning_rate': 0.039998136}
I0605 08:29:04.605577 139701982308224 model_lib_v2.py:700] {'Loss/classification_loss': 0.121629156,
 'Loss/localization_loss': 0.16370133,
 'Loss/regularization_loss': 0.2080817,
 'Loss/total_loss': 0.4934122,
 'learning_rate': 0.039998136}

Evaluate Model (Optional)

model_main_tf2.py

  • checkpoint_dir: the directory to read checkpoint.

Execute code:

python model_main_tf2.py \
--model_dir=training_configs/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8 \
--pipeline_config_path=training_configs/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/pipeline.config \
--checkpoint_dir=training_configs/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/

Execution results: calculate AP and AR

Export Model

exporter_main_v2.py

  • input_type: image_tensor
  • pipeline_config_path: pipeline.config path
  • trained_checkpoint_dir: the path to store checkpoint
  • output_directory: exported model path

Execute code:

!python exporter_main_v2.py \
--input_type image_tensor \
--pipeline_config_path training_configs/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/pipeline.config \
--trained_checkpoint_dir training_configs/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/  \
--output_directory exported-models/my_model_6000steps

Execution results:

INFO:tensorflow:Assets written to: exported-models/my_model_6000steps/saved_model/assets
I0605 09:07:21.034602 139745385867136 builder_impl.py:775] Assets written to: exported-models/my_model_6000steps/saved_model/assets
INFO:tensorflow:Writing pipeline config file to exported-models/my_model_6000steps/pipeline.config
I0605 09:07:22.310333 139745385867136 config_util.py:254] Writing pipeline config file to exported-models/my_model_6000steps/pipeline.config
Waymo
├───models
├───training_configs 
│   └───ssd_resnet50_v1_fpn_640x640_coco17_tpu-8
│       └─pipeline.config # create pipeline.config
├───pre-trained-models 
│   └───ssd_resnet50_v1_fpn_640x640_coco17_tpu-8
│       ├─ checkpoint/
│       ├─ saved_model/
│       └─ pipeline.config
├───exported-models 
│   └───my_model_6000steps
└───data
    ├───processed
    │   └─segment-???.tfrecord
    └───segment-???.tfrecord

Use model to predict

detect.py

  • saved_model_path: exported model path
  • test_path: image path
  • output_path: output predicted image path
  • min_score_thresh: confidience

Execute code:

!python detect.py \
--saved_model_path=exported-models/my_model_6000steps \
--test_path=test_image \
--output_path=output_image \
--min_score_thresh=.1

Execution results:

Reference

  1. LevinJ/tf_obj_detection_api
  2. Waymo Open Dataset
  3. Waymo quick start tutorial
  4. Tensorflow Object Detection API Tutorial
Deep Learning
Waymo
Object Detection
TensorFlow

--

--

許嘉倫
許嘉倫

Written by 許嘉倫

4 Followers
6 Following

目前是交大資訊科學與工程研究所碩士生

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams