ValueError: number sections must be larger than 0

[SeolhwaLee]

Hi,

I tried to apply OPACUS in my model. But I've got this error when I ran the code like below.

Traceback (most recent call last):
  File "/home/seol/miniconda3/envs/pytorch_p37/lib/python3.7/site-packages/numpy/lib/shape_base.py", line 772, in array_split
    Nsections = len(indices_or_sections) + 1
TypeError: object of type 'int' has no len()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main_dp.py", line 462, in <module>
    main(args)
  File "main_dp.py", line 166, in main
    for step, batch in enumerate(tqdm(memory_safe_data_loader)):
  File "/home/seol/miniconda3/envs/pytorch_p37/lib/python3.7/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/home/seol/miniconda3/envs/pytorch_p37/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/seol/miniconda3/envs/pytorch_p37/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 560, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "/home/seol/miniconda3/envs/pytorch_p37/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 512, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/home/seol/miniconda3/envs/pytorch_p37/lib/python3.7/site-packages/opacus/utils/batch_memory_manager.py", line 48, in __iter__
    batch_idxs, math.ceil(len(batch_idxs) / self.max_batch_size)
  File "<__array_function__ internals>", line 6, in array_split
  File "/home/seol/miniconda3/envs/pytorch_p37/lib/python3.7/site-packages/numpy/lib/shape_base.py", line 778, in array_split
    raise ValueError('number sections must be larger than 0.')
ValueError: number sections must be larger than 0.

One of my hack approaches was worked well without error when I added the below code in opacus/utils/batch_memory_manager.py for avoiding this error.

def __iter__(self):
     for batch_idxs in self.samplers:
         if not bool(batch_idxs):
             continue

But, the generated result was quite weird so I'm suspicious this hack was affecting model performance maybe?

Could you give any feedback that this hack is okay??

Thank you in advance!

P.S. I used this tutorial https://opacus.ai/tutorials/building_text_classifier (But different dataset and model)

[karthikprasad]

Hello! Thanks for raising the issue. I'll take a look at get back to you on this.

[karthikprasad]

Hi @SeolhwaLee. Are you using a different sampler that is resulting in an empty batch? If yes, is that intentional?

[SeolhwaLee]

@karthikprasad I used RandomSampler like this.

train_sampler = RandomSampler(train_dataset)
train_dataloader = DataLoader(
      dataset=train_dataset,
      sampler=train_sampler,
      collate_fn=train_dataset.collate_fn,
      batch_size=args.batch_size)

The empty batch is not my intent. Maybe it needs somehow raise empty batch process logic on this code?

[karthikprasad]

Hi @SeolhwaLee , The tutorial you linked does not use RandomSampler. When you call make_private() and pass your dataloader, Opacus internally switches it with DPDataLoader that uses Uniform Batch Sampler to sample according to SGM. Also note that the DPDataloader automatically handles empty_batches.

Since you are explicitly using RandomSampler, I suspect you are trying out something different? Could you share your code notebook and context?

[ashkan-software]

Hello @SeolhwaLee,

Has your issue been resolved?

[SeolhwaLee]

Hi @ashkan-software

I don't have time to investigate this recently. But will do it ASAP.

I will comment here if have any progress.