Rethinking FUN: Frequency-Domain Utilization Network

The search for efficient neural network architectures has gained much focus in recent years, where modern architectures focus not only on accuracy but also on inference time and model size. Here, we present FUN, a family of novel Frequency-domain Utilization Networks. These networks utilize the inherent efficiency of the frequency-domain by working directly in that domain, represented with the Discrete Cosine Transform. Using modern techniques and building blocks such as compound-scaling and inverted-residual layers we generate a set of such networks allowing one to balance between size, latency and accuracy while outperforming competing RGB-based models. Extensive evaluations verifies that our networks present strong alternatives to previous approaches. Moreover, we show that working in frequency domain allows for dynamic compression of the input at inference time without any explicit change to the architecture.

Description

Official PyTorch implementation of "Rethinking FUN: Frequency-domain Utilization Networks".

Based on the implementation by Ross Wightman

Recent Updates

Getting Started

Installation

• Clone this repo:

git clone https://github.com/kfir99/FUN.git
cd FUN

• Install a conda environment

conda create -n torch-env
conda activate torch-env
conda install -c pytorch pytorch torchvision cudatoolkit=10.2 jpeg2dct
conda install pyyaml

Pretrained Models

Pretrained models are available for download

Training

Preparing your data

Training and validation data should be organized in the following structure:

* data_dir
    * train
        * class_name_a
            * images
        * class_name_b
            * images
            
            .
        
            .
        
            .
    * validation
        * class_name_a
            * images
        * class_name_b
            * images
        
            .
        
            .
        
            .

Command line

./distributed_train.sh \
 <number of abailable GPUs>\
 <data_dir>\
 --output <desired output path>\
 --dct\ 
 --model <efun/efun_l/efun_s/efun_s_plus>\
 --drop-path <0.2/0.2/0.2/0.3>\
 --no-prefetcher\
 -b 128\
 --sched step\
 --epochs 450\
 --decay-epochs 2.4\
 --decay-rate .97\
 --opt rmsproptf\
 --opt-eps .001\
 -j 8\
 --warmup-lr 1e-6\
 --weight-decay 1e-5\
 --drop 0.2\ 
 --model-ema\
 --model-ema-decay 0.9999\ 
 --remode pixel\
 --reprob 0.2\
 --lr .048

Testing

Preparing your data

Training and validation data should be organized in the following structure:

* data_dir
    * class_name_a
        * images
    * class_name_b
        * images
        
        .
    
        .
    
        .

Command line

python validate.py\
 <path to test data>\
 --model <efun/efun_l/efun_s/efun_s_plus>\
 --checkpoint <path to trained model>\
 --dct\
 --no-prefetcher\
 -j 32\
 -b 256

Repository structure

Files added specifically for eFUN are marked in bold

Path	Description
FUN	Repository root folder
├  convert	Folder containing code for conversion from MXNET weights
├  docs	Folder containing documentation of the results and graphs use in the repository
├  notebooks	Folder with runnable notebooks for experimenting with the dataset
│  └  eFUN_inference_playground.ipynb	A basic notebook to run inference on some dataset using a pretrained eFUN model
├  results	Folder with reported results for original TIMM repository models
│  tests	Folder with some tests for original TIMM repository models
│  timm	Folder with actual TIMM pip package modules
│  ├  data	Folder with implementation of all data-related functions
│  └  rgb2dct.py	implementation of conversion from RGB image to its DCT representation
│  ├  loss	Folder with implementations of different loss functions
│  ├  models	Folder with implementation of all models provided in this repository
│  └  eFUN.py	Implementation of all eFUN models
│  ├  optim	Folder with implementations of different optimizers
│  ├  scheduler	Folder with implementations of different training schedulers
│  ├  utils	Folder with implementations of utility functions

TODOs

• [ ] sota-bench integration

Citation