notarize icon indicating copy to clipboard operation
notarize copied to clipboard
verified publisher icon electron

Notarization for arm64 version of app fails randomly but succeeds on retry

Open praneetloke opened this issue 1 year ago • 16 comments

I have an Electron app that is built and notarized using electron-builder. Notarization for the arm64 version of the app fails randomly without any additional information. The only log I have is this:

...
electron-notarize:spawn spawning cmd: xcrun args: [
  'notarytool',
  'submit',
  '/var/folders/5q/zcc7tkt11f13mhczs1yfd4qc0000gn/T/electron-notarize-PsiiWM/****.zip',
  '--apple-id',
  '*********',
  '--password',
  '*********',
  '--team-id',
  '*********',
  '--wait',
  '--output-format',
  'json'
] opts: {}
...
...
electron-notarize:spawn cmd xcrun terminated with code: null
electron-notarize:helpers work failed
  ⨯ Failed to notarize via notarytool.  Failed with unexpected result: 

  failedTask=build stackTrace=Error: Failed to notarize via notarytool.  Failed with unexpected result: 
                                                                                 at /Users/runner/work/****/****/node_modules/@electron/notarize/src/notarytool.ts:115:13
    at Generator.next (<anonymous>)
    at fulfilled (/Users/runner/work/****/****/node_modules/@electron/notarize/lib/notarytool.js:28:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
From previous event:
    at processImmediate (node:internal/timers:483:21)
From previous event:
    at readDirectoryAndSign (/Users/runner/work/****/****/node_modules/app-builder-lib/src/macPackager.ts:496:29)
    at MacPackager.signApp (/Users/runner/work/****/****/node_modules/app-builder-lib/src/macPackager.ts:506:11)
    at MacPackager.doSignAfterPack (/Users/runner/work/****/****/node_modules/app-builder-lib/src/platformPackager.ts:346:21)
    at MacPackager.doPack (/Users/runner/work/****/****/node_modules/app-builder-lib/src/platformPackager.ts:331:7)
    at MacPackager.pack (/Users/runner/work/****/****/node_modules/app-builder-lib/src/macPackager.ts:215:9)
    at Packager.doBuild (/Users/runner/work/****/****/node_modules/app-builder-lib/src/packager.ts:459:9)
    at executeFinally (/Users/runner/work/****/****/node_modules/builder-util/src/promise.ts:12:14)
    at Packager.build (/Users/runner/work/****/****/node_modules/app-builder-lib/src/packager.ts:393:31)
    at executeFinally (/Users/runner/work/****/****/node_modules/builder-util/src/promise.ts:12:14)

There is no other information about the failure. In a successful execution, the exit code is 0 which is expected. I have set DEBUG=electron-notarize* as per the README in this repo but it doesn't seem to help with emitting any additional information about the failure. Any help/tips on diagnosing this would be appreciated.

EDIT Jan 2025: See https://github.com/electron/notarize/issues/219#issuecomment-2523261208. If you are using GitHub Actions:

  • and have noticed that the build fails randomly despite having accepted all pending Apple Terms
  • and are using macos-latest
  • downgrade your runner to macos-13.
    • macos-13 runner is an Intel-based runner with 14GB RAM.

praneetloke avatar Dec 03 '24 16:12 praneetloke

Hey @praneetloke, thanks for the report. Does this also repro with a basic repo? Also could you include the version of notarize that you're using?

VerteDinde avatar Dec 03 '24 22:12 VerteDinde

@VerteDinde just to add to this, I'm also having a very similar issue. The error output is identical to the one reported, but in my case, the problematic architecture is x86, arm64 notarizes fine.

The notarization had been working fine for a long time but suddenly it stopped working, but I couldn't figure out exactly when it happened because the last build was from a couple months ago, so no logs for that.

Thought it could be related to GitHub upgrading their macos-latest runner to use macOS 14, but that happened at the start of 2024 and notarization was definitely working after that.

I'm running the latest version of notarize: 2.5.0.

fempyrean avatar Dec 04 '24 12:12 fempyrean

Hello @VerteDinde, thank you for your response.

Does this also repro with a basic repo?

Unfortunately, I do not know if that's the case since the failure is sporadic right now. I might have an entire day of zero failures.

Also could you include the version of notarize that you're using?

I use [email protected] which in turn uses @electron/[email protected].

To add additional context, my CI, like @fempyrean, also runs on GH actions. I use macos-latest for the macOS builds which runs macOS 14 arm64 runners. I have a suspicion that macOS is perhaps terminating xcrun and hence the lack of a proper exit code (it's null)? However, I don't know of a way to prove this; it's just a theory. At this point, I am not sure where the issue lies.

These failures started about two months ago with version @electron/[email protected] that we had been running since the beginning of this year. Interestingly, the stack trace was slightly different with that version and also the failure was flip-flopping between x64 and arm64 with the following stack trace:

electron-notarize:spawn spawning cmd: xcrun args: [
  'notarytool',
  'submit',
  '/var/folders/g6/rgtlsw6n123b0gt5483s5_cm0000gn/T/electron-notarize-ajOoEA/***.zip',
  '--apple-id',
  '*********',
  '--password',
  '*********',
  '--team-id',
  '*********',
  '--wait',
  '--output-format',
  'json'
] opts: {}
electron-notarize:spawn cmd xcrun terminated with code: null <----NOTE: null exit code.
electron-notarize:helpers work failed
  ⨯ Unexpected end of JSON input  failedTask=build stackTrace=SyntaxError: Unexpected end of JSON input
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /Users/runner/work/***/node_modules/@electron/notarize/src/notarytool.ts:79:25
    at Generator.next (<anonymous>)
    at fulfilled (/Users/runner/work/***/node_modules/@electron/notarize/lib/notarytool.js:28:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
From previous event:
    at processImmediate (node:internal/timers:478:21)
From previous event:
    at readDirectoryAndSign (/Users/runner/work/***/node_modules/app-builder-lib/src/macPackager.ts:478:29)
    at MacPackager.signApp (/Users/runner/work/***/node_modules/app-builder-lib/src/macPackager.ts:488:11)
    at MacPackager.doSignAfterPack (/Users/runner/work/***/node_modules/app-builder-lib/src/platformPackager.ts:336:21)
    at MacPackager.doPack (/Users/runner/work/***/node_modules/app-builder-lib/src/platformPackager.ts:321:7)
    at MacPackager.pack (/Users/runner/work/***/node_modules/app-builder-lib/src/macPackager.ts:198:9)
    at Packager.doBuild (/Users/runner/work/***/node_modules/app-builder-lib/src/packager.ts:445:9)
    at executeFinally (/Users/runner/work/***/node_modules/builder-util/src/promise.ts:12:14)
    at Packager._build (/Users/runner/work/***/node_modules/app-builder-lib/src/packager.ts:379:31)
    at Packager.build (/Users/runner/work/***/node_modules/app-builder-lib/src/packager.ts:340:12)
    at executeFinally (/Users/runner/work/***/node_modules/builder-util/src/promise.ts:12:14)

About three weeks ago, I updated to @electron/[email protected] and the failure still occurs but only inarm64 now. Although I am not sure if that's just a coincidence.

praneetloke avatar Dec 04 '24 16:12 praneetloke

I'm having this issue also, and came across this reference: https://github.com/actions/runner-images/issues/9811

I've added this step to the action I'm using, and running it a few times to see if it stabilizes:

      
      - name: xcode-select
        if: matrix.os == 'macos-latest'
        run: sudo xcode-select -s /Library/Developer/CommandLineTools

edit: womp womp.

2024-12-06T13:30:29.517Z electron-notarize:spawn cmd xcrun terminated with code: null
2024-12-06T13:30:29.517Z electron-notarize:helpers work failed
  ⨯ Failed to notarize via notarytool.  Failed with unexpected result:

final edit: I think @praneetloke's suspicion about the xcrun crash is spot on. i found some other references yesterday that suggested changing back to macos-13 in the action runner, and I did that. The process is stable now, so I'm leaving it at that and moving on.

bradfair avatar Dec 06 '24 13:12 bradfair

@bradfair

i found some other references yesterday that suggested changing back to macos-13 in the action runner, and I did that. The process is stable now, so I'm leaving it at that and moving on.

That's a great idea!

Also I took a look at GH's hosted runners and their macos-13 runners are Intel-based with 14GB RAM. Their M1 runners are half of that at 7GB. My guess is xcrun is terminated as a result of low memory sometimes. Though, again, I don't have a way to prove this right now.

praneetloke avatar Dec 06 '24 15:12 praneetloke

Same thing just happened here on macos latest:

https://github.com/mifi/lossless-cut/actions/runs/12211950166/job/34069829185 I've had a scheduled job run every day and I believe today is the first time i've seen this error. also macos-latest (needed for apple silicon builds). Maybe we could impleemtn a retry?

Edit: it still happens intermittently. last week:

  • https://github.com/mifi/lossless-cut/actions/runs/12765242934/job/35579071342
  • https://github.com/mifi/lossless-cut/actions/runs/12732484808/job/35487735589
  • https://github.com/mifi/lossless-cut/actions/runs/12707306521/job/35421953916
  • https://github.com/mifi/lossless-cut/actions/runs/12649279112/job/35245346751

mifi avatar Dec 07 '24 10:12 mifi

@mifi it looks like you are using electron-builder, it supports cross-platform compilation for arm64 on Intel-based macOS. However, if this was the first time you've seen a problem, you might want to check if you've got a pending Apple terms agreement waiting to be accepted in your Apple Developer Account. (I got one yesterday that I accepted.) If you've already accepted that, it would be helpful if you could observe your builds for a few days at least to see how often it fails with macos-latest, if at all.

praneetloke avatar Dec 07 '24 15:12 praneetloke

got a pending Apple terms agreement waiting to be accepted in your Apple Developer Account.

That seems like it might have been the problem. I also got pending terms/agreements that day. Now it seems to be succeeding. Will keep a look at the job going forward

mifi avatar Dec 11 '24 06:12 mifi

I encountered the exact same problem. This issue is unrelated to the Apple terms agreement. All terms in my Apple developer account have been accepted, yet I still encounter this problem.

P.S. If I manually use xcrun notarytool to submit my app, the notarization is immediately accepted.

Hawstein avatar Jan 06 '25 15:01 Hawstein

macos-13 runner however is significantly slower. Takes 2 minutes longer for my whole pipeline (5min on macos-latest vs 7min on macos-13)

codeofandrin avatar Jan 12 '25 20:01 codeofandrin

Having this issue as well on macos-latest when using electron-forge. Trying to build a universal app, and it seems about 50% of the time it will just randomly fail. I've accepted the Apple terms so it isn't that.

fredluetkemeier avatar Jan 15 '25 02:01 fredluetkemeier

Same here, but on x64 (macos-latest) as well:

2025-01-18T09:58:24.162Z electron-osx-sign Application signed.
✖ Finalizing package [FAILED: Failed to notarize via notarytool.  Failed with unexpected result: ]
✖ Packaging for x64 on darwin [FAILED: Failed to notarize via notarytool.  Failed with unexpected result: ]
✖ Packaging application [FAILED: Failed to notarize via notarytool.  Failed with unexpected result: ]
✖ Running package command [FAILED: Failed to notarize via notarytool.  Failed with unexpected result: ]
✖ Running make command [FAILED: Failed to notarize via notarytool.  Failed with unexpected result: ]

An unhandled rejection has occurred inside Forge:
Error: Failed to notarize via notarytool.  Failed with unexpected result:
at /Users/runner/work/…/…/app/node_modules/@electron/notarize/lib/notarytool.js:141:23
    at Generator.next (<anonymous>)
    at fulfilled (/Users/runner/work/…/…/app/node_modules/@electron/notarize/lib/notarytool.js:28:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
node:child_process:957
    throw err;
    ^

Error: Command failed: npm run publish -- --arch=x64
    at genericNodeError (node:internal/errors:983:15)
    at wrappedFn (node:internal/errors:537:14)
    at checkExecSyncError (node:child_process:882:11)
    at execSync (node:child_process:954:15)
    at Object.<anonymous> (/Users/runner/work/…/…/build.js:198:2)
    at Module._compile (node:internal/modules/cjs/loader:1562:14)
    at Object..js (node:internal/modules/cjs/loader:1699:10)
    at Module.load (node:internal/modules/cjs/loader:1313:32)
    at Function._load (node:internal/modules/cjs/loader:1123:12)
    at TracingChannel.traceSync (node:diagnostics_channel:322:14) {
  status: 1,
  signal: null,
  output: [ null, null, null ],
  pid: 4234,
  stdout: null,
  stderr: null
}

Node.js v22.13.0

thomasbachem avatar Jan 18 '25 11:01 thomasbachem

Btw, GitHub released macos-15 into public preview, but it has the same issue.

macos-13 works for me.

thomasbachem avatar Jan 18 '25 13:01 thomasbachem

we are the team from LobeHub and building lobe-chat now with electron. electron-notarize help us to make notarization and it's great. Thanks a lot!

But recently we also meet the same issue:

  • https://github.com/lobehub/lobe-chat/actions/runs/15154151200/job/42605740442 is failed
  • https://github.com/lobehub/lobe-chat/actions/runs/15157287619/job/42615145911 is successful

I've accepted the Apple terms so maybe there is a bug?

arvinxx avatar May 21 '25 09:05 arvinxx

I too am having issues with it just hanging trying to notarize after signing via github actions. It'll timeout (after 17 minute timeout I put) but kick it right off again and it'll finish fine in 8 minutes.

jamesrusso avatar Jul 31 '25 16:07 jamesrusso

Is there any update to this? The suggested fix is to use macos-13, however that is closing down in December 2025.

https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/

rekkyrosso avatar Sep 22 '25 13:09 rekkyrosso