google
Use FileStore in FlickrPhotosImporter
similar to #612 - looks like the FlickrPhotosImporter still uses datastore and if someone has too many photos or albums, this causes an exception.
com.google.cloud.datastore.DatastoreException: The value of property "org.datatransferproject.spi.transfer.types.TempPhotosData" is longer than 1500 bytes.
I0318 09:13:29.088563 at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:128)
I0318 09:13:29.088890 at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:154)
I0318 09:13:29.088898 at com.google.cloud.datastore.DatastoreImpl$4.call(DatastoreImpl.java:496)
I0318 09:13:29.088903 at com.google.cloud.datastore.DatastoreImpl$4.call(DatastoreImpl.java:493)
I0318 09:13:29.088905 at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
I0318 09:13:29.088916 at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
I0318 09:13:29.088962 at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
I0318 09:13:29.088999 at com.google.cloud.datastore.DatastoreImpl.commit(DatastoreImpl.java:492)
I0318 09:13:29.089147 at com.google.cloud.datastore.TransactionImpl.commit(TransactionImpl.java:108)
I0318 09:13:29.089154 at com.google.dataliberation.transfer.worker.extensions.cloud.T2XJobStore.update(T2XJobStore.java:274)
I0318 09:13:29.089234 at org.datatransferproject.datatransfer.flickr.photos.FlickrPhotosImporter.importItem(FlickrPhotosImporter.java:108)
I0318 09:13:29.089242 at org.datatransferproject.datatransfer.flickr.photos.FlickrPhotosImporter.importItem(FlickrPhotosImporter.java:49)
I0318 09:13:29.089349 at org.datatransferproject.transfer.CallableImporter.call(CallableImporter.java:48)
I0318 09:13:29.089420 at org.datatransferproject.transfer.CallableImporter.call(CallableImporter.java:30)
I0318 09:13:29.089590 at org.datatransferproject.types.transfer.retry.RetryingCallable.call(RetryingCallable.java:66)
I0318 09:13:29.089652 at org.datatransferproject.transfer.PortabilityInMemoryDataCopier.copyHelper(PortabilityInMemoryDataCopier.java:121)
I0318 09:13:29.089668 at org.datatransferproject.transfer.PortabilityInMemoryDataCopier.copy(PortabilityInMemoryDataCopier.java:71)
I0318 09:13:29.089676 at org.datatransferproject.transfer.JobProcessor.processJob(JobProcessor.java:109)
I0318 09:13:29.089677 at org.datatransferproject.transfer.Worker.doWork(Worker.java:33)
I0318 09:13:29.089679 at org.datatransferproject.transfer.WorkerMain.poll(WorkerMain.java:142)
I0318 09:13:29.089723 at org.datatransferproject.transfer.WorkerMain.main(WorkerMain.java:68)
I0318 09:13:29.092681 Caused by: com.google.datastore.v1.client.DatastoreException: The value of property "org.datatransferproject.spi.transfer.types.TempPhotosData" is longer than 1500 bytes., code=INVALID_ARGUMENT
I0318 09:13:29.092772 at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:226)
I0318 09:13:29.092778 at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:279)
I0318 09:13:29.092781 at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:186)
I0318 09:13:29.092784 at com.google.datastore.v1.client.Datastore.commit(Datastore.java:87)
I0318 09:13:29.092787 at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:152)
Looks like there are several places where we use Jobstore.create(UUID jobId, String Key, T model) instead of JobStore.Create(UUID jobId, String key, InputStream stream) - in the GoogleJobStore, the first uses datastore (resulting in the above exception) and the second uses filestore, we should probably just modify the first to use the correct thing
Example is MicrosoftPhotosImporter.To make the switch we would need each DataModel type to support conversion to and from InputStream (we do this manually in the GooglePhotosExporter in converJsonToInputStream() and use the ObjectMapper to convert from the stream to the TempPhotosData)