Milan Nosáľ

OK, I'll need a bit more context if you don't mind.. right now I'm using the following approach to use MLXLLM that's based on the swift example code (a bit...

AFAIK, blocking resigning isActive is not possible, so the only way how not to crash is cancel/pause anything that is schedule to happen as soon as I get the notification...

Alright, I quickly put some prototype together in https://github.com/MilanNosal/mlx-swift-examples/pull/1 Basically I "sprinkled" `load`, `generate` and `prepare` with `Task.checkCancellation()`... I gotta go to bed now, as it's late in my timezone,...

Hey @davidkoski ! Can I have hopefully a quick question? > but one of the calls, eval(model) can potentially take several seconds. Perhaps it could iterate over the parameters in...

Thanks for the long explanation! I'll give it a bit more thought and tests over the weekend.

ok, a follow up question: before generating from input, I'm calling: `let input = try await context.processor.prepare(input: input)` where in the callstack `prepare` -> `step` -> `convertToToken` -> `processor?.didSample(token: y)`...

hm, OK, what's making it worse is that the time period is 1.5 second for that given prompt, the longer the prompt the bigger the time period.. which sucks, as...

cool, thanks.. minor update - by setting the smaller `prefillStepSize` on the parameters it stopped crashing.. I guess that gives it more time between steps to cancel it?

Hey @deet ! Do you intend to submit a PR with the cancellation, or are you only researching the issue? Looks like I need to take a look at this...