This is the official repository for the WACV paper CookGAN: Meal Image Synthesis from Ingredients. The code is tested with Python 3.8, PyTorch 1.6, CUDA 10.2 on Ubuntu 18.04

Prepare Dataset

Dowload original dataset

Download Recipe1M dataset from http://pic2recipe.csail.mit.edu/, make sure you have download and unzip all images and the files det_ingrs.json, layer1.json, layer2.json. Your data folder should look like the one shown below

CookGAN/data/Recipe1M/
    images/
        train/
        val/
        test/
    recipe1M/
        det_ingrs.json
        layer1.json
        layer2.json

Create an environment using Python 3.8, install the required packages.

pip install -r requirements.txt

Simplify dataset

run python clean_recipes_with_canonical_ingrs.py to generate ./data/Recipe1M/recipes_withImage.json which just contains the simplified recipes with images (N=402760), e.g.

{
    "id": "00003a70b1",
    "url": "http://www.food.com/recipe/crunchy-onion-potato-bake-479149",
    "partition": "test",
    "title": "Crunchy Onion Potato Bake",
    "instructions": [
      "Preheat oven to 350 degrees Fahrenheit.",
      "Spray pan with non stick cooking spray.",
      "Heat milk, water and butter to boiling; stir in contents of both pouches of potatoes; let stand one minute.",
      "Stir in corn.",
      "Spoon half the potato mixture in pan.",
      "Sprinkle half each of cheese and onions; top with remaining potatoes.",
      "Sprinkle with remaining cheese and onions.",
      "Bake 10 to 15 minutes until cheese is melted.",
      "Enjoy !"
    ],
    "ingredients": [
      "milk",
      "water",
      "butter",
      "mashed potatoes",
      "whole kernel corn",
      "cheddar cheese",
      "French - fried onions"
    ],
    "valid": [
      true,
      true,
      true,
      true,
      true,
      true,
      true
    ],
    "images": [
      "3/e/2/3/3e233001e2.jpg",
      "7/f/7/4/7f749987f9.jpg",
      "a/a/f/6/aaf6b2dcd3.jpg"
    ]
  }

Train Models

All models (except word2vec) could be monitored using wandb.

Train word2vec

Go to retrieval_model and run python train_word2vec.py to generate models/word2vec_recipes.bin.

Pre-train UPMC-Food-101 classifier

Go to ./pretrain_upmc, follow ./pretrain_upmc/README to pretrain the image encoder on UPMC-Food-101 dataset.

Train the attention-based retrieval model

Run

CUDA_VISIBLE_DEVICES=0 bash run_retrieval.sh

to train the attention-based recipe retrieval model. Here, 010 means only using ingredients to train the model. The code also supports training using all three domains by --text_info=111 (title+ingredients+instructions).

Train CookGAN

Go to CookGAN/cookgan and run

CUDA_VISIBLE_DEVICES=0 bash run.sh

to train CookGAN on salad.

Test Models

go to CookGAN/metrics/,

• Update the configurations following configs/salad+cookgan.yaml.
• Run python calc_inception.py to generate statistics for real images.
• Run python fid.py to compute the FIDs under a certain checkpoint directory.
• Run python medR.py to compute the median ranks under a certain checkpoint directory.

Genearte an image from the pre-trained model

1. Download the trained model from the Google drive folder.
2. Run the notebook test_model.ipynb to generate an image.

License

MIT