App Hanging -[TSLocationManager queue:type:]

[DigifloHendrix]

Your Environment

• Plugin version: 4.16.2 (react-native-background-geolocation)
• Platform: iOS
• OS version: 15.8.2
• Device manufacturer / model: iPhone8,1
• React Native version: 0.73.6
• Plugin config:

{
  enableHeadless: true,
  disableLocationAuthorizationAlert: false,
  locationAuthorizationRequest: 'Always',
  backgroundPermissionRationale: {
    title: "Allow *** to access to this device's location in the background?",
    message: 'In order to track your activity in the background and to make sure you get job offers in your area, please enable "{backgroundPermissionOptionLabel}" location permission',
    positiveAction: 'Change to {backgroundPermissionOptionLabel}',
    negativeAction: 'Cancel',
  },
  desiredAccuracy: BackgroundGeolocation.DESIRED_ACCURACY_NAVIGATION,
  distanceFilter: 10,
  activityType: BackgroundGeolocation.ACTIVITY_TYPE_AUTOMOTIVE_NAVIGATION,
  stopTimeout: 5,
  debug: false,
  logLevel: __DEV__ ? BackgroundGeolocation.LOG_LEVEL_INFO : BackgroundGeolocation.LOG_LEVEL_ERROR,
  stopOnTerminate: true,
  startOnBoot: false,
  url: `${getApiBase()}/devices/telemetry/sync`,
  method: 'POST',
  batchSync: false,
  autoSync: false,
  preventSuspend: true,
  heartbeatInterval: 60,
  notification: {
    title: 'Background tracking engaged',
    text: '*** is tracking your current location to ensure you get job offers in your area',
  },
}

Expected Behavior

The background geolocation should work consistently without hanging.

Actual Behavior

The app is hanging for at least 2000 ms when processing location updates.

Steps to Reproduce

1. Initialize the GeolocationService
2. Start location tracking
3. Receive location updates
4. Attempt to process and sync telemetry data to the server

Context

The app is using react-native-background-geolocation for tracking location in the background. The issue occurs when processing location updates and attempting to sync telemetry data. The app seems to be hanging in the queue:type: method of TSLocationManager.

Debug logs

Thread Stack Trace

Thread 0:

* Occurred in non-app
  libsystem_kernel.dylib
  __ulock_wait + 0x1688
  (2 hidden frames)

* libsystem_platform.dylib
  _os_unfair_lock_lock_slow + 0x1760

* libobjc.A.dylib
  objc_sync_enter + 0x51b8

* *** (In App)
  -[TSLocationManager queue:type:] + 0xa98e34

* *** (In App)
  -[TSLocationManager onUpdateCurrentPosition:location:type:] + 0xa94da4

* *** (In App)
  __49-[TSLocationManager createLocationChangedHandler]_block_invoke + 0xa94824

* *** (In App)
  -[LocationManager locationManager:didUpdateLocations:] + 0xaad478

* Called from CoreLocation
  CLClientStopVehicleHeadingUpdates + 0x2b548
  (10 hidden frames)

* CoreLocation
  CLCopyAppsUsingLocation + 0xde90

* CoreLocation
  CLCopyTechnologiesInUse + 0xa0a4

* LocationSupport
  <redacted> + 0x2384

* CoreFoundation
  __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 0x6d688

* CoreFoundation
  __CFRunLoopDoBlocks + 0x6e4ac

* CoreFoundation
  __CFRunLoopRun + 0xb1d8

* CoreFoundation
  CFRunLoopRunSpecific + 0x1e170

* GraphicsServices
  GSEventRunModal + 0x1984

* UIKitCore
  -[UIApplication _run] + 0x4e5a84

* UIKitCore
  UIApplicationMain + 0x27ef74

* *** (In App)
  main + 0x72f0

* Called from unknown
  <redacted> at 0x103c1c4d0

Thread 23:

* Occurred in non-app
  libsystem_kernel
  +0x001688
  __ulock_wait
  (4 hidden frames)

* libdispatch
  +0x004d5c
  _dlock_wait

* libdispatch
  +0x004b48
  _dispatch_thread_event_wait_slow$VARIANT$mp

* libdispatch
  +0x011bf8
  __DISPATCH_WAIT_FOR_QUEUE__

* libdispatch
  +0x01181c
  _dispatch_sync_f_slow

* *** (In App)
  +0xabfc44
  -[TSQueue runOnMainQueueWithoutDeadlocking:]

* *** (In App)
  +0xa98f60
  -[TSLocationManager queue:type:]

* *** (In App)
  +0xa908b8
  -[TSLocationManager getCurrentPosition:]

* *** (In App)
  +0x1a111c
  -[RNBackgroundGeolocation getCurrentPosition:success:failure:]

* Called from CoreFoundation
  +0x020cec
  __invoking___
  (2 hidden frames)

* CoreFoundation
  +0x03cdd4
  -[NSInvocation invoke]

* CoreFoundation
  +0x071208
  -[NSInvocation invokeWithTarget:]

* *** (In App)
  +0x2752f4
  -[RCTModuleMethod invokeWithBridge:module:arguments:]

* *** (In App)
  +0x2773f8
  facebook::react::invokeInner

* *** (In App)
  +0x277048
  facebook::react::RCTNativeModule::invoke

* Called from libdispatch
  +0x063090
  _dispatch_call_block_and_release
  (5 hidden frames)

* libdispatch
  +0x064090
  _dispatch_client_callout

* libdispatch
  +0x00a738
  _dispatch_lane_serial_drain$VARIANT$mp

* libdispatch
  +0x00b1f0
  _dispatch_lane_invoke$VARIANT$mp

* libdispatch
  +0x014ec4
  _dispatch_workloop_worker_thread

* libsystem_pthread
  +0x001dfc
  _pthread_wqthread

[christocracy]

Let me know how to reproduce it.

I believe the problem is likely in your own code due to use of await somewhere.

the plug-in is very careful to execute its actions in non-blocking background-threads.

[DigifloHendrix]

Thank you for your quick response. You're right that the issue might be in our implementation. After reviewing our code, I've identified some areas where we're using await that could potentially cause blocking behavior. Here's a more detailed breakdown of our implementation:

1. We're using await in our onLocation callback:

const onLocation: Subscription = BackgroundGeolocation.onLocation(
    async (location: Location) => {
        BackgroundGeolocation.startBackgroundTask().then(
            async (taskId: any) => {
                try {
                    // ... (some code omitted for brevity)
                    
                    // Save the telemetry to Realm
                    await realmManager.performWithRealm(realm => {
                        realm.write(() => {
                            realm.create(
                                'Telemetry',
                                {
                                    id: location.uuid,
                                    data: JSON.stringify(telemetryData),
                                    synced: false,
                                },
                                UpdateMode.All,
                            );
                        });
                    });

                    // Process the telemetry queue
                    await this.processTelemetryQueue();
                } catch (error) {
                    // ... (error handling)
                } finally {
                    await BackgroundGeolocation.stopBackgroundTask(taskId);
                }
            },
        );
    },
);

2. Our processTelemetryQueue method also uses await:

private async processTelemetryQueue() {
    await this.acquireProcessingLock('processTelemetryQueue', async () => {
        try {
            const isConnected = await isInternetReachable();

            if (isConnected) {
                // ... (processing logic)
                await this.processTelemetryBatch(telemetryItems);
            }
        } catch (error) {
            // ... (error handling)
        }
    });
}

3. We're also using await in our getCurrentPosition method:

public getCurrentPosition = async (
    options: CurrentPositionRequest,
): Promise<Location | undefined> => {
    if (this._enabled) {
        return await BackgroundGeolocation.getCurrentPosition(options);
    }
    return undefined;
};

To reproduce the issue:

1. Initialize our GeolocationService
2. Start location tracking
3. The app receives location updates frequently (every few seconds)
4. Each update triggers the onLocation callback, which starts a background task
5. The background task saves data to Realm and attempts to process the telemetry queue
6. If internet is available, it tries to sync the data

The hang occurs during this process, specifically in the queue:type: method of TSLocationManager.

Given your feedback, it seems our use of await in these areas might be causing the main thread to block. Do you have any recommendations on how we should restructure our code to avoid this issue? Should we be handling these operations differently to ensure they don't block the main thread?

We appreciate any guidance you can provide to help us resolve this issue.

I'd like to provide some context on why we've structured our code this way:

1. Metadata Requirements: We need to attach specific metadata to each telemetry point. This metadata includes information such as the current job status, device information, and other contextual data that's critical for our business logic. This metadata can change between location updates, so we need to ensure it's captured at the exact moment of the location update.

2. Offline Functionality: Our app needs to function in offline scenarios. When the device goes offline, we continue to collect location data along with the associated metadata. This data is stored locally using Realm.

3. Historical Context: When the device comes back online, we need to sync not just the current location, but all the stored offline data. Each of these historical data points needs to maintain its original metadata. If we were to only use current metadata when syncing, we'd lose the historical context of each location update, which is crucial for our tracking requirements.

4. Data Integrity: By processing each location update synchronously (using await), we ensure that the metadata associated with each location point is accurate for that specific moment in time. This helps maintain data integrity across our system.

5. Complex Sync Logic: Our processTelemetryQueue method needs to handle various scenarios, including partial syncs, retries, and ensuring data is synced in the correct order. This complexity led us to implement it as an async function.

We understand that this approach might not be optimal for performance, especially given the frequent location updates. We're open to suggestions on how to restructure our code to maintain these requirements while improving performance and avoiding potential hangs.

Some questions we have:

1. Is there a recommended way to attach custom metadata to each location update without blocking the main thread?
2. How can we ensure that offline data is properly stored and synced with its original metadata when the device comes back online?
3. Are there best practices for handling complex, stateful operations (like our sync logic) in the context of background geolocation?

We're eager to improve our implementation and would greatly appreciate any insights or recommendations you can provide based on these specific requirements.

[Szymon20000]

Hello @christocracy ! Thank you for helping us here. I really appreciate it. Looks like the code that was sent is no up to date.

I don't have access to the TSLocation class as it's already compiled but unless you wait there for message form js with a lock until it happens I don't see how await could cause any deadlock. The BackgroundLocation native module which is bridge based without any synchronous calls. When you send event to Js or resolve a promise it happens asynchronously.

We have 2 appHang events that took over 2s (It doesn't necessarily mean it was a deadlock) it could be just operation on the main thread that made UI thread unresponsive for over 2s. In both cases looks like TSLocation gets "didUpdateLocations" event from he os (on the main thread)

And another thread (TSQueue) is executing getCurrentPosition call. (asynchronous from what I see)

[christocracy]

Create for me a simple “hello-world” app which reproduces this. Share it with me in a public GitHub repo

[swng-henk]

Create for me a simple “hello-world” app which reproduces this. Share it with me in a public GitHub repo

We will try. At the moment we are not sure what exact sequence of events causes the issue. Our theory is that the problem is caused by a call to getCurrentPosition() while we also have an onLocation() listener.

[christocracy]

Nobody has reported this in 9 years. I believe it’s your own code blocking the UI thread.

[swng-henk]

Nobody has reported this in 9 years. I believe it’s your own code blocking the UI thread.

Thanks. Just to clarify: does this mean it should be OK to call getCurrentPosition() while an onLocation() listener is active?

[Szymon20000]

@christocracy If you check the stack trace of the main thread (First stacktrace) you see that the only thing on the main thread is the TSLocationManager code. If there is a deadlock that is caused by us that would mean that TSLocationManager on the main thread is waiting for a resource that we locked. What that could be?

Could you share a code of this method [TSLocationManager queue:type:]?

[Szymon20000]

It's also possible that there is no deadlock at all and that method for some weird reason takes over 2s.

[christocracy]

Start disabling your code until the problem goes away.

[DigifloHendrix]

@christocracy, thank you for your input. We've analyzed the situation further:

1. The hang occurs within TSLocationManager queue:type:, which is part of the plugin's compiled code.
2. Main thread stack trace shows only TSLocationManager code during the hang.
3. Issue appears when onLocation() listener is active and getCurrentPosition() is called concurrently.
4. We're working on a minimal reproduction app, but the intermittent nature makes it challenging.
5. Can you review the queue:type: method for potential thread safety issues?
6. Are there additional diagnostics we can enable in the plugin for more detailed information?
7. Is it safe to call getCurrentPosition() while an onLocation() listener is active?

We're open to further investigation on our end. Please advise on specific areas to examine or modify for testing.

[christocracy]

Why would you call .getCurrentPosition in your onLocation event-handler?? That would create an infinite loop.

[christocracy]

Your analysis of the native code is a red herring. There’s nothing wrong with the native code.

[DigifloHendrix]

@christocracy, thank you for your follow-up. To clarify:

1. We're not calling getCurrentPosition within the onLocation handler. These are separate operations in our app.
2. getCurrentPosition is a method provided by your library that we use independently of the onLocation events.
3. Our usage:
   • We have onLocation set up to receive regular updates.
   • Separately, we occasionally call getCurrentPosition for immediate location data.
4. The hang occurs during normal operation, not in an infinite loop scenario.
5. Metadata handling: We need to attach custom metadata to each location update. This metadata can change between updates and is crucial for our offline functionality and historical context. How can we efficiently attach this metadata without risking performance issues or crashes?

Can you suggest best practices for using getCurrentPosition alongside onLocation in your library? We want to ensure we're not inadvertently creating conflicts.

[christocracy]

Can you suggest best practices for using getCurrentPosition alongside onLocation in your library?

Don’t. Ever. .getCurrentPosition CAUSES .onLocation events to be fired. Why would you request a new location in an event-handler which just provided you a location?!

[DigifloHendrix]

@christocracy, I apologize for any confusion. Let me clarify our situation:

1. We are NOT calling getCurrentPosition within the onLocation handler. These are separate, independent operations in our app.
2. The issue occurs during normal operation where:
   • We have onLocation set up for regular updates.
   • Separately and occasionally, we call getCurrentPosition for immediate location data.
3. We've identified a potential deadlock in your library:
   • One thread is waiting on the UI thread using dispatch_sync_f_slow
   • The UI thread is then trying to acquire a lock with objc_sync_enter
   • This lock seems to be already held by the first thread, causing a deadlock
4. Stack traces for reference: Thread 1: libdispatch +0x1181c _dispatch_sync_f_slow -> <***> +0xabfc44 -[TSQueue runOnMainQueueWithoutDeadlocking:] Main thread: libobjc.A +0x51b8 objc_sync_enter -> <***> +0xa98e34 -[TSLocationManager queue:type:]
5. Our main concerns remain:
   • Efficiently attaching custom metadata to each location update
   • Resolving crashes originating from the transistorsoft library

Could you please advise on: a) Best practices for using getCurrentPosition alongside onLocation without conflicts b) Efficient methods for attaching custom metadata to location updates c) Strategies to investigate and resolve the crashes we're experiencing

[christocracy]

As I said, nobody has reported a problem like this in 9 years. Create for me a simple hello-world app which reproduces this issue. Share it with me in a public GitHub repo.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[Technickster]

Sorry to bring up closed issue.

We also see this, it is quite intermittent as well so we have not been able to reproduce in a debug environment. I have been doing some testing and trying to reproduce it but haven't been able to unfortunately. @DigifloHendrix did you figure it out?

It is seen in about 5-10% of our user base. App Hang TSLocationManager The app was terminated while unresponsive 4 days ago – 2 months ago

[andrecrimb]

I'm also facing this problem 👍🏽

[andrecrimb]

Upgrading to the latest version 4.17.4 solved the issue for me (I was previously using 4.15.2).