Offline map support on public branch for testing

[brnkhy]

Hey everyone! I have been working on offline map support (https://docs.mapbox.com/help/troubleshooting/mobile-offline/) for a while now and wanted to let you know it's on a public branch for testing at the moment.

You can find it here; https://github.com/mapbox/mapbox-unity-sdk/tree/offlineMaps

And usage goes something like this;

 var region = new OfflineRegion(
            MapName,
            _minLatLng,
            _maxLatLng,
            _minZoom,
            _maxZoom,
            elevationTilesetId,
            imageryTilesetId,
            vectorTilesetId);
var response = MapboxAccess.Instance.OfflineManager.CreateOfflineMap(region.Name, region);

This will download and cache all tiles in this region in given zoom range. I just pushed it today so there might be bugs and issues of course. Just to be clear, this doesn't mean you can ship data with your application or anything. This is just a variation of regular ambient cache. Regular ambient cache time out, recycle out etc. Offline maps adds a flag to tiles so they won't time out like regular ambient cache tiles. So it ensures those tiles will be in the cache.

In example, let's say you're building a hiking app where people go off grid frequently to hike. You can let them predownload and cache their hiking area in advance, at their home when they have connectivity for example. SDK will ensure cached tiles will not be removed/recycled/deleted so when they go hiking and don't have connection, their tile data will still be in the cache.

This feature also comes with a lot of changes in Sqlite management so please let me know if you notice any bugs and/or performance issues regarding data in general.

[brnkhy]

for anyone trying this out, please let me know if you run into troubles, it's super beta so there will be bugs but I want to finish this asap so any fixes to this will be priority for me 🙏

[Markovicho]

first note:

OfflineMapDemoController.cs remove using directive for:

using UnityEditor.Rendering.LookDev;

it seems like it's not used and will throw errors if you are not using the right pipeline: Assets\Mapbox\Examples\9_OfflineMap\Scripts\OfflineMapDemoController.cs(7,29): error CS0234: The type or namespace name 'LookDev' does not exist in the namespace 'UnityEditor.Rendering' (are you missing an assembly reference?)

by the way thx for the afford. This feature is pretty important for a lot of showcases. Looking forward to check this out

Postscript:

Running on Unity 2019.4.5f LTS on hitting play will result in those errors. It's running on a german system which could possibly cause some formatting/encoding issues (?)

[brnkhy]

Hey @Markovicho , Removed the using directive, I have no idea it was there really. Thanks!

Second issue is happening because your existing old SQLite table doesn't work with the new offline maps code. If you use Mapbox -> Clear File Cache from the top menu, it'll delete the existing file and recreate it using new system. Then you shouldn't get the error.

[Markovicho]

Thx! clearing the file cache worked.

postscript:

Building for Android will run into another unused using directive:

CachingWebFileSource.cs ==> using Unity.UNetWeaver

using Unity.UNetWeaver; causing: Assets\Mapbox\Core\mapbox-sdk-cs\Platform\Cache\CachingWebFileSource.cs(1,13): error CS0234: The type or namespace name 'UNetWeaver' does not exist in the namespace 'Unity' (are you missing an assembly reference?)

Test on PC:

Configuration:

minZoom: 10 maxZoom: 18

Zoom while Downloading: 16

After the download completed i started to zoom out/in. But only zoomLevel 15+16 seems to work offline oO

I attached a videocapture. there is a little textfield which is showing the current zoomLevel for debug purposes

https://drive.google.com/file/d/1kGao-LXjOX_Sao7f_dcpvcrX1MgdVlgk/view?usp=sharing

[brnkhy]

@Markovicho I'm not sure what's up with those unused directives, great catch!

I'll check the zoom thing but I think it might be something about overzooming (16+ levels). I can't remember for sure now but I might intentionally or not might have capped it at z16. I'll check and update asap. Thanks for the video by the way, it's great to see it's working well otherwise!

[brnkhy]

@Markovicho I think I fixed the zoom issue. The problem was with the demo scene, it used satellite imagery but sample code downloaded streets imagery so when you went offline, it wasn't able to find different zoom levels of satellite. Somehow, z15-16 was cached though, probably with regular ambient caching.

Also found and fixed a bug with cache expiration date. Thanks a lot!

[Markovicho]

This seems to be solved and is working now as expected :-) (Tested on Windows and Android 11)

How is the feature behaving on entering the following situations:

• Display standby : ==> Unity-Application will go to sleep...
• Switching to another application ==> Unity-App will go to sleep as well here ...

This could be an interesting point regarding larger downloads or slow internet connections. I assume it would be a large task to outsource this functionality into an native plugin to avoid pausing the download (right ?) Otherwise we need to find some way to keep the user in the foreground application while the download is running (UX-Design needs to be aware of that point)

[brnkhy]

@Markovicho awesome, thanks a lot with the help, I really appreciate it! As I said I want to get this done as soon as possible.

I'm afraid I have no idea about the issue you described though. I don't do mobile stuff much so I'm not sure how it works but I suspect OS freezes the app when it's in the background? Or doing tricky something in the background without app developers knowledge might not be preferable either. Either way I think it's safer to leave that to app developer. I'm not sure but as far as I know other Mapbox SDKs doesn't do this either right?

[Markovicho]

Like you said no matter if you're using Android or iOS, the application will freeze completely. The only way to keep functionality active in this case is to write an separate native plugin for both plattforms which will continue the tasks while the unity app is freezed. Maybe thats even more work than the current solution itself. As long as the download will continue correctly after returning to the app thats not a big deal and more an question of the app design. Anyways this feature is such a big improvement for the unity sdk. thx in advance !!

[raggnic]

Nice ! Is it ok with the mapbox data licence tough?

[brnkhy]

@raggnic it's the same offline map support other Mapbox SDKs provide. Why did you thought that might be an issue? Am I missing something? You got me worried :)

[raggnic]

Ok sorry I may have read the intro too fast. I was not aware of the statement you've made, about shipping an app with included data. It may be super useful to us but we're still on a fork from 1.45.... so long way to go

[brnkhy]

@raggnic I highly suggest updating but if it'll be too much work, I'm hoping to release (raster) performance work public soon as well (1-2 weeks) so you might want to wait for that.

[JackMoljerc]

It works!! Is there any way to change the third-party tile Servers ,such as my own local geoserver?

[Markovicho]

How about the releaseplan ? Is it worth to use this branch or should we wait for an upcoming official release of the mapbox unity sdk ?

[brnkhy]

@JackMoljerc that would be a different feature and I probably won't have a chance to look into that for a while. Sorry.

@Markovicho I think I'll wait on this for now and release it together with performance branch. Both works developed separately and they both customize sqlite database so to prevent more issues, I'll just try to update it all in once. I'm afraid it won't be trivial though so might take some time. I really want to finish all this before 2021 though, so this isn't official or anything but I want raster performance improvements + offline maps + ton of bug fixes in before 2021.

[brnkhy]

@Markovicho I started working on that merge I mentioned yesterday already but admittedly I found a few points would use some improvement. Even some possible bugs. So I think it would be wiser not to use it in final product for a little longer.

[Markovicho]

@brnkhy sure we will wait for the final release. The mentioned changes in your last post (raster performance etc) sounds pretty much like what we need to get a stable release ready version. Is there a public branch for testing the raster performance ? maybe we could help improve that process by some testing feedback :-)

[brnkhy]

@Markovicho I want to release it as soon as possible (for a while now) for exactly that reason. But it's such a big change that I want to release it at least in a decent state. I thought I was there really, I had a reliable version but then I realized merging that with offline maps won't be trivial and again cause a lot of changes so I decided to get that out of way first. I'll definitely let you all know when it's ready though, I can really use some help/feedback 🙏

[Kundankmr500]

@brnkhy I have downloaded this branch to use the offline map feature but when I open the OfflineMap scene there is OfflineMapListPrefab prefab reference is missing in OfflineMapControl Gameobject. please have a look at the screenshot. Can you tell me the OfflineMapListPrefab Prefab name and Path?

[LauraLaureus]

Hello @Kundankmr500 did you manage to get any advances on this line ?

[brnkhy]

Hey @Kundankmr500 @LauraLaureus I'm sorry I just saw this. Turns out that prefabs was indeed missing 🤦‍♂️ I'm really sorry for that. I pushed he prefab on same branch. I suspect you still might have issues as it was crated using 2019.4 and previous versions might not detect it properly. It's OfflineMapListItem under offline maps example folder. It's used only for the demo and shouldn't effect the feature itself.

fix commit; https://github.com/mapbox/mapbox-unity-sdk/commit/1039e4053fbe4a5ab65635b527a817cf2da59346

[LauraLaureus]

Thank you @brnkhy ! Now it works :) Can you guide me, or at least put me in the right track, on how to load the offline maps ?

[alexandre-gcc]

Can you guide me, or at least put me in the right track, on how to load the offline maps ?

Yes, that would be nice please 😉

[nilsk123]

Hey @Kundankmr500 @LauraLaureus I'm sorry I just saw this. Turns out that prefabs was indeed missing 🤦‍♂️ I'm really sorry for that. I pushed he prefab on same branch. I suspect you still might have issues as it was crated using 2019.4 and previous versions might not detect it properly. It's OfflineMapListItem under offline maps example folder. It's used only for the demo and shouldn't effect the feature itself.

fix commit; 1039e40

Hey hey, we're super excited for the performance and offline improvements. Any telling whether or not that will be done in the next few months?

[Markovicho]

Hey @Kundankmr500 @LauraLaureus I'm sorry I just saw this. Turns out that prefabs was indeed missing 🤦‍♂️ I'm really sorry for that. I pushed he prefab on same branch. I suspect you still might have issues as it was crated using 2019.4 and previous versions might not detect it properly. It's OfflineMapListItem under offline maps example folder. It's used only for the demo and shouldn't effect the feature itself. fix commit; 1039e40

Hey hey, we're super excited for the performance and offline improvements. Any telling whether or not that will be done in the next few months?

Would be really great to know. Actually we are waiting for this feature for a long time now which started very promising . Now we are running into the situation that we need to launch our application including those big performance issues. Maybe Covid-19 is causing the standby. The last official release was dated 2019 (https://docs.mapbox.com/mapbox-unity-sdk/docs/05-changelog.html). But at least @brnkhy is giving some implications that mapbox-unity is not a dead framework. I hope thats still the case.

Would be great to get a little outlook on the next release !

[brnkhy]

@nilsk123 @Markovicho really sorry for the long wait! We are reviewing the performance branch for sensitive information at the moment (api keys, dev info etc) and then I'll push it on this public repo right away. It will be a limited version though as I removed bunch of features like preview mode, some vector layer stuff.... oh and most likely AR.

First public branch will be just image performance improvements (elevation included). I'm pretty confident in the performance improvements, only slight concern would be the size of changes and bunch of removed features.

I'm working on vector layer at the moment and it's much more difficult. Still, again by limiting the feature set, I'm hoping to push it here as soon as possible.

At that point I'll probably just work on the public branches so you'll have access to latest stuff.

Just to make it clear, this is not a sdk new version release. That requires a lot of extra work like packaging, testing etc. This'll just be a branch for now but it's being used for a big application at the moment so it's not just an unreliable playground or anything either.

[Markovicho]

Thx for the detailed answer @brnkhy Really appreciate that.

I'm pretty confident in the performance improvements, only slight concern would be the size of changes and bunch of > removed features.

In the end we are currently skipping basic features like zooming / panning as well as avoiding any tile loading which is not totally necassary. Otherwise it's hard to create a reliable User-Experience on mobile. So i think it would be pretty easy to get over some extended concepts or convenience tools :-) The only feature which would be a showstopper for ous would be dropping the rendering of 3D-Buildings. But i think that won't be the case (?)

[alexandre-gcc]

@nilsk123 @Markovicho

really sorry for the long wait!

We are reviewing the performance branch for sensitive information at the moment (api keys, dev info etc) and then I'll push it on this public repo right away.

It will be a limited version though as I removed bunch of features like preview mode, some vector layer stuff.... oh and most likely AR.

First public branch will be just image performance improvements (elevation included). I'm pretty confident in the performance improvements, only slight concern would be the size of changes and bunch of removed features.

I'm working on vector layer at the moment and it's much more difficult. Still, again by limiting the feature set, I'm hoping to push it here as soon as possible.

At that point I'll probably just work on the public branches so you'll have access to latest stuff.

Just to make it clear, this is not a sdk new version release. That requires a lot of extra work like packaging, testing etc.

This'll just be a branch for now but it's being used for a big application at the moment so it's not just an unreliable playground or anything either.

Hey, so the map will be optimised? This concerns only offline maps?

[brnkhy]

@nilsk123 @Markovicho @AlexandreAero performance branch is public now and you can find it here; https://github.com/mapbox/mapbox-unity-sdk/tree/performance

I'll try to write a proper thing for it as soon as possible but meanwhile you can find a short tweet thread about it here; https://twitter.com/brnkhy/status/1379845983190851587