safeplaces-dct-app icon indicating copy to clipboard operation
safeplaces-dct-app copied to clipboard

Google Import should expand logged points for duration

Open penrods opened this issue 6 years ago • 11 comments

During the import of Google data the time is imported a single time/coordinate. However, the Google data has durations, so we should use that to expand points. This will be useful later in the intersections.

Basically, the Google data contains something like:

[ 
 {"placeVisit": {
      "location" : {
        "latitudeE7" : 423936728,
        "longitudeE7" : -710802770,
      },
      "duration" : {
        "startTimestampMs" : "1583085476383",
        "endTimestampMs" : "1583086992817"
      },
   }
}
...
]

The start time is: Sunday, March 1, 2020 5:57:56.383 PM and the end time is: Sunday, March 1, 2020 6:23:12.817 PM

That is 26 minutes of the user at the same point, so import it in 5 minute intervals so it looks like: [ { startTimestampMs, latitudeE7, longitudeE7 }, { startTimestampMs + 5 minutes, latitudeE7, longitudeE7 }, { startTimestampMs + 10 minutes, latitudeE7, longitudeE7 }, { startTimestampMs + 15 minutes, latitudeE7, longitudeE7 }, { startTimestampMs + 20 minutes, latitudeE7, longitudeE7 }, { startTimestampMs + 25 minutes, latitudeE7, longitudeE7 }, ]

Don't stress too much about overlap between places, we can work with having time segments that are narrower than 5 minutes. But we will likely have issues intersecting in encrypted space with a timespan between points that is > 5 minutes.

Additionally, the transit points aren't being imported. It would be nice to import waypoint data.

penrods avatar Mar 16 '20 18:03 penrods

Also, please verify that the Google timestamps are in UTC. I fairly sure they are, but would like this verified.

penrods avatar Mar 16 '20 18:03 penrods

for each placeVisit we need to check 

if --> endTimestampMs - startTimestampMs is greater than 300000
then number of points = ( endTimestampMs - startTimestampMs ) / 300000
Finally Add these many points with adding 300000 to startTimestampMs of previous entry

(300000 is the difference for 5 mins)

harshvitra avatar Mar 25 '20 23:03 harshvitra

@penrods @harshvitra I'd like to grab this one if it's not currently being worked on

jordajm avatar Mar 26 '20 18:03 jordajm

It doesn't look like the unzipped file is getting saved to the directory specified in GetFileName():

LOG  Unzip Completed for /storage/emulated/0/Download and /storage/emulated/0/Download/takeout-20200327T014023Z-001.zip
LOG  ENOENT: /storage/emulated/0/Download/Takeout/Location History/Semantic Location History/2020/2020_MARCH.json (No such file or directory), open '/storage/emulated/0/Download/Takeout/Location History/Semantic Location History/2020/2020_MARCH.json' ENOENT

Should this be taken care of in this issue as well?

gabalmat avatar Mar 27 '20 03:03 gabalmat

It doesn't look like the unzipped file is getting saved to the directory specified in GetFileName():

LOG  Unzip Completed for /storage/emulated/0/Download and /storage/emulated/0/Download/takeout-20200327T014023Z-001.zip
LOG  ENOENT: /storage/emulated/0/Download/Takeout/Location History/Semantic Location History/2020/2020_MARCH.json (No such file or directory), open '/storage/emulated/0/Download/Takeout/Location History/Semantic Location History/2020/2020_MARCH.json' ENOENT

Should this be taken care of in this issue as well?

Looks like in Android, this is a situation where your Google takeout file does not have 2020_MARCH.json, this happened to be a couple of times with the extracts. Also in the code the file is defaulted to be of MARCH every time, which might be true initially but now should be expanded to be MARCH-and-beyond.

popofibo avatar Mar 30 '20 13:03 popofibo

I can take a look once #69 and #77 are addressed. Feel free to assign this one to me.

sergesemashko avatar Apr 07 '20 15:04 sergesemashko

@penrods Shall location rather be store with Start and end timestamps to reduce amount of logged data?

sergesemashko avatar Apr 17 '20 11:04 sergesemashko

I'm going to take a look at this issue

sergesemashko avatar Apr 23 '20 02:04 sergesemashko

placeVisit" : { "location" : { "latitudeE7" : 324914959, "longitudeE7" : -921784525, "placeId" : "ChIJbbNi6KM3LoYRvSDTmrO3fYY", "address" : "223 W Heights Dr\nWest Monroe, LA 71292\nUSA", "name" : "223 W Heights Dr", "sourceInfo" : { "deviceTag" : -1864277346 }, "locationConfidence" : 36.61622 }, "duration" : { "startTimestampMs" : "1607129593011", "endTimestampMs" : "1607130430990" },

manda545 avatar May 13 '21 02:05 manda545

I need this address translated into normal date an time.

manda545 avatar May 13 '21 02:05 manda545

Can anyone translate this or see what I’ve posted

manda545 avatar May 13 '21 17:05 manda545