Dragging only when holding tap on an area

[MarioUnlam]

I have a list which should be sortable. Each element has a button on the left. When the user holds the finger on that button, the item should be draggable, but not when tapping or holding anywhere else. I can't make this work.

I set manuallyActivateRows to true, and modified the plugin so it properly exposes the "setScrollEnabled" function. I put that function inside a TouchableX, and the element becomes active. But I cannot make it active AND drag at the same time. I need to tap first to activate it, and then drag the element again to change the order. I tried using PanHandlers with these parameters:

onPanResponderGrant: (evt, gestureState) => {
	if (active) return;
	toggleRowActive(evt, gestureState);
},
onPanResponderMove: (evt, gestureState) => {
},
onPanResponderTerminationRequest: (evt, gestureState) => {
	if (active) {
		return false;
	}

	return true;
},
onPanResponderRelease: (evt, gestureState) => {
	if (active) {
		toggleRowActive(evt, gestureState);
	}
},
onPanResponderTerminate: (evt, gestureState) => {
	if (active) {
		toggleRowActive(evt, gestureState);
	}
},

But it does the same. It activates when I tap, and deactivates when I tap again. Is there any way to accomplish what I want?