hexo icon indicating copy to clipboard operation
hexo copied to clipboard
verified publisher icon hexojs

Out of Memory on 33103 html and with image 671111 regular files

Open githubbbie opened this issue 1 year ago • 10 comments

Check List

  • [X] I have already read Docs page & Troubleshooting page.
  • [X] I have already searched existing issues and they are not help to me.
  • [X] I examined error or warning messages and it's difficult to solve.
  • [X] I am using the latest version of Hexo. (run hexo version to check)
  • [ ] My Node.js is matched the required version.

Expected behavior

Run:

hexo server

or

hexo generate

And the site is generated in record time :)

Actual behavior

Tried #2165 and no luck.

Get out of memory error as such:

Ok, trying on 33103 files and getting fail at 16 gb

#!/usr/bin/env node --max_old_space_size=16384

in node_modules/hexo-cli/bin/hexo

Error:

<--- Last few GCs --->

[137:0x7964cb02b000]    33039 ms: Scavenge (reduce) (interleaved) 3446.1 (3671.1) -> 3446.1 (3668.1) MB, pooled: 0 MB, 8.24 / 0.00 ms  (average mu = 0.274, current mu = 0.252) external memory pressure; 
[137:0x7964cb02b000]    33791 ms: Mark-Compact (reduce) 3446.3 (3668.1) -> 3444.6 (3648.6) MB, pooled: 0 MB, 96.37 / 0.00 ms  (+ 649.8 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 789 ms) (average mu = 0.262, 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

Aborted (core dumped)

AND, here is the error at 32gb

<--- Last few GCs --->

[148:0x7aed4cf6f000]    34166 ms: Scavenge (reduce) (interleaved) 3444.6 (3660.1) -> 3444.6 (3657.1) MB, pooled: 0 MB, 8.24 / 0.00 ms  (average mu = 0.262, current mu = 0.248) external memory pressure; 
[148:0x7aed4cf6f000]    34920 ms: Mark-Compact (reduce) 3444.7 (3657.1) -> 3443.1 (3638.9) MB, pooled: 0 MB, 99.86 / 0.00 ms  (+ 648.2 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 791 ms) (average mu = 0.256, 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

Aborted (core dumped)

How to reproduce?

  1. hexo generate

Is the problem still there under Safe mode?

Yes

Your Node.js & npm version

node -v && npm -v

v22.9.0
10.8.3

Your Hexo and Plugin version

/hexo # npm ls --depth 0
[email protected] /hexo
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
`-- [email protected]

Your package.json

{
  "name": "hexo-site",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "build": "hexo generate",
    "clean": "hexo clean",
    "deploy": "hexo deploy",
    "server": "hexo server"
  },
  "hexo": {
    "version": "7.3.0"
  },
  "dependencies": {
    "hexo": "^7.3.0",
    "hexo-bridge": "^2.0.3",
    "hexo-generator-archive": "^2.0.0",
    "hexo-generator-category": "^2.0.0",
    "hexo-generator-index": "^4.0.0",
    "hexo-generator-tag": "^2.0.0",
    "hexo-renderer-ejs": "^2.0.0",
    "hexo-renderer-marked": "^6.3.0",
    "hexo-renderer-stylus": "^3.0.1",
    "hexo-server": "^3.0.0",
    "hexo-theme-landscape": "^1.0.0"
  }
}

Your site's _config.yml (Optional)

No response

Others

No response

githubbbie avatar Sep 30 '24 20:09 githubbbie

Wow, got further this time:

node --max-old-space-size=5168 node_modules/hexo/bin/hexo generate -c 1

Template render error: (unknown path)
  Error: unexpected end of comment
    at Object._prettifyError (/hexo/node_modules/nunjucks/src/lib.js:32:11)
    at Template.render (/hexo/node_modules/nunjucks/src/environment.js:442:21)
    at Environment.renderString (/hexo/node_modules/nunjucks/src/environment.js:313:17)
    at /hexo/node_modules/hexo/dist/extend/tag.js:206:22
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.fromNode.Promise.fromCallback (/hexo/node_modules/bluebird/js/release/promise.js:209:30)
    at Tag.render (/hexo/node_modules/hexo/dist/extend/tag.js:205:35)
    at Object.onRenderEnd (/hexo/node_modules/hexo/dist/hexo/post.js:399:32)
    at /hexo/node_modules/hexo/dist/hexo/render.js:77:29
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/hexo/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/hexo/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/hexo/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/hexo/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/hexo/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/hexo/node_modules/bluebird/js/release/async.js:86:9)

Will up to 8192

node --max-old-space-size=8192 node_modules/hexo/bin/hexo generate -c 1

githubbbie avatar Sep 30 '24 22:09 githubbbie

node --max-old-space-size=8192 node_modules/hexo/bin/hexo generate -c 1

Same error

Template render error: (unknown path) Error: unexpected end of comment at Object._prettifyError (/hexo/node_modules/nunjucks/src/lib.js:32:11) at Template.render (/hexo/node_modules/nunjucks/src/environment.js:442:21) at Environment.renderString (/hexo/node_modules/nunjucks/src/environment.js:313:17) at /hexo/node_modules/hexo/dist/extend/tag.js:206:22 at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23) at Promise.fromNode.Promise.fromCallback (/hexo/node_modules/bluebird/js/release/promise.js:209:30) at Tag.render (/hexo/node_modules/hexo/dist/extend/tag.js:205:35) at Object.onRenderEnd (/hexo/node_modules/hexo/dist/hexo/post.js:399:32) at /hexo/node_modules/hexo/dist/hexo/render.js:77:29 at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/hexo/node_modules/bluebird/js/release/promise.js:547:31) at Promise._settlePromise (/hexo/node_modules/bluebird/js/release/promise.js:604:18) at Promise._settlePromise0 (/hexo/node_modules/bluebird/js/release/promise.js:649:10) at Promise._settlePromises (/hexo/node_modules/bluebird/js/release/promise.js:729:18) at _drainQueueStep (/hexo/node_modules/bluebird/js/release/async.js:93:12) at _drainQueue (/hexo/node_modules/bluebird/js/release/async.js:86:9)

githubbbie avatar Oct 01 '24 13:10 githubbbie

ANDDDD

time node --max-old-space-size=16384 node_modules/hexo/bin/hexo generate -c 1

FATAL Something's wrong. Maybe you can find the solution here: https://hexo.io/docs/troubleshooting.html
Template render error: (unknown path)
  Error: unexpected end of comment
    at Object._prettifyError (/hexo/node_modules/nunjucks/src/lib.js:32:11)
    at Template.render (/hexo/node_modules/nunjucks/src/environment.js:442:21)
    at Environment.renderString (/hexo/node_modules/nunjucks/src/environment.js:313:17)
    at /hexo/node_modules/hexo/dist/extend/tag.js:206:22
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.fromNode.Promise.fromCallback (/hexo/node_modules/bluebird/js/release/promise.js:209:30)
    at Tag.render (/hexo/node_modules/hexo/dist/extend/tag.js:205:35)
    at Object.onRenderEnd (/hexo/node_modules/hexo/dist/hexo/post.js:399:32)
    at /hexo/node_modules/hexo/dist/hexo/render.js:77:29
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/hexo/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/hexo/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/hexo/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/hexo/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/hexo/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/hexo/node_modules/bluebird/js/release/async.js:86:9)
Command exited with non-zero status 2
real    1h 13m 13s
user    1h 13m 22s
sys     0m 23.78s

Would like to use HEXO and find a solution as I believe in the nodejs path, but seems like when others have hit this wall, they just move to HUGO and it just works.

Any help here?

githubbbie avatar Oct 01 '24 15:10 githubbbie

Ok, surely 32 GB of Ram is enough? haha this must be a deeeper issue...

time node --max-old-space-size=32768 node_modules/hexo/bin/hexo generate -c 1

FATAL Something's wrong. Maybe you can find the solution here: https://hexo.io/docs/troubleshooting.html
Template render error: (unknown path)
  Error: unexpected end of comment
    at Object._prettifyError (/hexo/node_modules/nunjucks/src/lib.js:32:11)
    at Template.render (/hexo/node_modules/nunjucks/src/environment.js:442:21)
    at Environment.renderString (/hexo/node_modules/nunjucks/src/environment.js:313:17)
    at /hexo/node_modules/hexo/dist/extend/tag.js:206:22
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.fromNode.Promise.fromCallback (/hexo/node_modules/bluebird/js/release/promise.js:209:30)
    at Tag.render (/hexo/node_modules/hexo/dist/extend/tag.js:205:35)
    at Object.onRenderEnd (/hexo/node_modules/hexo/dist/hexo/post.js:399:32)
    at /hexo/node_modules/hexo/dist/hexo/render.js:77:29
    at tryCatcher (/hexo/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/hexo/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/hexo/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/hexo/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/hexo/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/hexo/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/hexo/node_modules/bluebird/js/release/async.js:86:9)
Command exited with non-zero status 2
real    1h 8m 35s
user    1h 8m 42s
sys     0m 24.28s

githubbbie avatar Oct 01 '24 23:10 githubbbie

we need separate generator. Like:

  • hexo generate tags only generate tags page
  • hexo generate post only generate post
  • hexo generate page only generate page (non post in source/_posts)
  • hexo generate categories only generate categories page
  • hexo generate assets only process scss, js, images, non markdown files, post assets into public folder

This memory heap error because hexo handle all items inside array json database.

for now marked, markdown-it to process single markdown file, consume 200MB RAM. hexo need migration.

hexo cannot process when the posts or assets more than device memory can handle.

Rather than process all items in one process, why not get -> process -> save to public folder of each file, it will make hexo memory friendly.

dimaslanjaka avatar Oct 20 '24 12:10 dimaslanjaka

Wow!

On October 20, 2024 7:39:11 AM CDT, Dimas Lanjaka @.***> wrote:

for now marked, markdown-it to process single markdown file, consume 200MB RAM. hexo need migration.

-- Reply to this email directly or view it on GitHub: https://github.com/hexojs/hexo/issues/5552#issuecomment-2424907489 You are receiving this because you authored the thread.

Message ID: @.***>

githubbbie avatar Oct 21 '24 02:10 githubbbie

Nice idea. Who are the active devs on this project?

On October 20, 2024 7:29:12 AM CDT, Dimas Lanjaka @.***> wrote:

we need separate generator. Like:

  • hexo generate tags only generate tags page
  • hexo generate post only generate post
  • hexo generate page only generate page (non post in source/_posts)
  • hexo generate categories only generate categories page
  • hexo generate assets only process scss, js, images, non markdown files, post assets into public folder

This memory heap error because hexo handle all items inside array json database.

-- Reply to this email directly or view it on GitHub: https://github.com/hexojs/hexo/issues/5552#issuecomment-2424903692 You are receiving this because you authored the thread.

Message ID: @.***>

githubbbie avatar Oct 21 '24 02:10 githubbbie

Look at my debugger, main out of memory occurs while rendering markdown files

image

markdown-it now taking more RAM usage

image

dimaslanjaka avatar Oct 24 '24 22:10 dimaslanjaka

Wow! Whonare the developers on this project?

Starting to feel like this project is abandoned

On October 24, 2024 6:00:17 PM CDT, Dimas Lanjaka @.***> wrote:

Look at my debugger, main out of memory caused while rendering markdown files

image

-- Reply to this email directly or view it on GitHub: https://github.com/hexojs/hexo/issues/5552#issuecomment-2436484408 You are receiving this because you authored the thread.

Message ID: @.***>

githubbbie avatar Oct 26 '24 00:10 githubbbie

Wow! Whonare the developers on this project? Starting to feel like this project is abandoned … On October 24, 2024 6:00:17 PM CDT, Dimas Lanjaka @.> wrote: Look at my debugger, main out of memory caused while rendering markdown files image -- Reply to this email directly or view it on GitHub: #5552 (comment) You are receiving this because you authored the thread. Message ID: @.>

i dont know who the starter dev. but, these ppl still active to review PR @uiolee @SukkaW

dimaslanjaka avatar Oct 26 '24 08:10 dimaslanjaka