mt-captioning

This repository corresponds to the PyTorch implementation of the paper Multimodal Transformer with Multi-View Visual Representation for Image Captioning. By using the bottom-up-attention visual features (with slight improvement), our single-view Multimodal Transformer model (MT_sv) delivers 130.9 CIDEr on the Kapathy's test split of MSCOCO dataset. Please check our paper for details.

Table of Contents

0. Prerequisites
1. Training
2. Testing

Prerequisites

Requirements

• Python 3
• PyTorch >= 1.4.0
• Cuda >= 9.2 and cuDNN

The annotation files can be downloaded here and unzipped to the datasets folder.

The visual features are extracted by our bottom-up-attention.pytorch repo using the following scripts:

# 1.extract the bbox from the image
$ python3 extract_features.py --mode caffe \
          --config-file configs/bua-caffe/extract-bua-caffe-r101-bbox-only.yaml \
          --image-dir <image_dir> --out-dir <bbox_dir> --resume

# 2. extract the roi feature by bbox
$ python3 extract_features.py --mode caffe \
         --config-file configs/bua-caffe/extract-bua-caffe-r101-gt-bbox.yaml \
         --image-dir <image_dir> --gt-bbox-dir <bbox_dir> --out-dir <output_dir> --resume

We provided a pre-extracted features in the datasets/mscoco/features/val2014 folder for the image in datasets/mscoco/image to help validating the correctness of the extracted features.

We use the ResNet-101 as our backbone and extract features for the MSCOCO dataset to the datasets/mscoco/features/frcn-r101 folder.

Finally, the datasets folder will have the following structure:

|-- datasets
   |-- mscoco
   |  |-- features
   |  |  |-- frcn-r101
   |  |  |  |-- train2014
   |  |  |  |  |-- COCO_train2014_....jpg.npz
   |  |  |  |-- val2014
   |  |  |  |  |-- COCO_val2014_....jpg.npz
   |  |  |  |-- test2015
   |  |  |  |  |-- COCO_test2015_....jpg.npz
   |  |-- annotations
   |  |  |-- coco-train-idxs.p
   |  |  |-- coco-train-words.p
   |  |  |-- cocotalk_label.h5
   |  |  |-- cocotalk.json
   |  |  |-- vocab.json
   |  |  |-- glove_embeding.npy

Training

The following script will train a model with cross-entropy loss :

$ python train.py --caption_model svbase --ckpt_path <checkpoint_dir> --gpu_id 0

1. caption_model refers to the model while been trained, such as svbase and umv

2. ckpt_path refers to the dir to save checkpoint.

3. gpu_id refers to the gpu id.

Based on the model trained with cross-entropy loss, the following script will load the pre-trained model and then fine-tune the model with self-critical loss:

$ python train.py --caption_model svbase --learning_rate 1e-5 --ckpt_path <checkpoint_dir> --start_from <checkpoint_dir_rl> --gpu_id 0 --max_epochs 25

1. caption_model refers to the model while been trained.

2. learning_rate refers to the learning rate use in self-critical.

3. ckpt_path refers to the dir to save checkpoint.

4. gpu_id refers to the gpu id.

Testing

Given the trained model, the following script will report the performance on the val split of MSCOCO:

$ python test.py --ckpt_path <checkpoint_dir> --gpu_id 0

1. ckpt_path refers to the dir to save checkpoint.

2. gpu_id refers to the gpu id.

Pre-trained models

We provided the pre-trained model for the single-view MT model at present. More models will be added in the future.

Citation

If this repository is helpful for your research, we'd really appreciate it if you could cite the following paper:

@article{yu2019multimodal,
  title={Multimodal transformer with multi-view visual representation for image captioning},
  author={Yu, Jun and Li, Jing and Yu, Zhou and Huang, Qingming},
  journal={IEEE Transactions on Circuits and Systems for Video Technology},
  year={2019},
  publisher={IEEE}
}

Acknowledgement

We thank Ruotian Luo for his self-critical.pytorch, cider and coco-caption repos.