data-transfer-project icon indicating copy to clipboard operation
data-transfer-project copied to clipboard

Add perUserRate limit to more services that need it.

Open bwillard opened this issue 6 years ago • 2 comments

After: https://github.com/google/data-transfer-project/pull/693 any service that needs to can add a per user rate limter to their service.

bwillard avatar Apr 22 '19 15:04 bwillard

i would like to contribute for this project. Can you guys help me out on how to start and proceed with this issue ?

umeshm2000 avatar Aug 04 '19 18:08 umeshm2000

Sure thing, so I think there are roughly 4 things that need to be done per service:

  1. Figure out what the per-user rate limit should be, it could be that a service doesn't have one, in which case it is still probibly worth it to document that in step 2.
  2. Add a <serviceName>.yaml file similar to: https://github.com/google/data-transfer-project/blob/master/extensions/data-transfer/portability-data-transfer-deezer/src/main/resources/config/deezer.yaml
  3. In the transfer extension do something like: 'TransferServiceConfig transferServiceConfig = context.getService(TransferServiceConfig.class);' like: https://github.com/google/data-transfer-project/blob/a277d3020d13117483cf88a661c75c112e3ea354/extensions/data-transfer/portability-data-transfer-deezer/src/main/java/org/datatransferproject/transfer/deezer/DeezerTransferExtension.java#L75
  4. In the importer/exporter for the service get a RateLimiter and then call acquire for each call to the service's api, like in: https://github.com/google/data-transfer-project/blob/master/extensions/data-transfer/portability-data-transfer-deezer/src/main/java/org/datatransferproject/transfer/deezer/DeezerApi.java

bwillard avatar Aug 05 '19 12:08 bwillard