[RFC] 3.0 Updates - Parallel Building and Downloading

[crazywhalecc]

[RFC] 3.0 Updates - Parallel Building and Downloading

static-php-cli 2.x has downloader for all sources, but downloading them one by one is quite slow.

It's best to support parallel downloads by default:

1. Calculating download artifacts
2. Queue download sources
3. Use fiber to call curl or git

For building, it's more complicated than downloading, since it depends on the dependency tree structure calculated by builder input, and make command always use -j$(nproc) to use all cores simuteniously, we may only reduce several minutes when parallel configuring.

If you have any question or have better suggestions, please leave comment here with quote.

[henderkes]

For downloading yes, for sure. For building it will not increase build times, only save a bit of time for ./configure commands. I think it's not worth it, because it makes build errors less reproducible.

[crazywhalecc]

I'm working on parallel downloading, but it's a little tricky. Introducing an event loop would simplify things considerably, I don't think it's necessary to add dependencies though

[henderkes]

We could use swoole or uv for this. People who don't install the extensions can use the prebuilt spc binary, after all.

[jingjingxyk]

config 目录下的配置信息，组成行为树，遍历行为树的节点，每个树节点执行命令，并记录执行过程 和 结果到节点

让多条命令 并行 或者 并发 运行，可从如下角度进行选取

1. 多进程
2. 多线程
3. 协程
4. 队列

需要解决： 线程间、进程间通信

控制运行逻辑：

1. 遍历行为树的节点
2. 行为树的节点使用状态机控制命令执行

如果不限制编程语言，可选项可多了，如果限制在 PHP 编程语言，swoole workerman swow 等 实现事件循环的框架

[boonkerz]

parallel extension is also an option

[crazywhalecc]

Okay. Good news: we can make shell execution like coroutine using fiber pool with very small changes and without any extensions or dependencies !

 // fill pool
while (count($fiber_pool) < $pool_count && ($artifact = array_shift($this->artifacts)) !== null) {
    $current = $count - count($this->artifacts);
    $fiber = new \Fiber(function () use ($artifact, $current, $count) {
        $this->downloadWithType($artifact, $current, $count, true);
    });
    $fiber->start();
    $fiber_pool[] = $fiber;
}
// check pool
foreach ($fiber_pool as $index => $fiber) {
    if ($fiber->isTerminated()) {
        // remove from pool
        unset($fiber_pool[$index]);
        ++$downloaded;
        InteractiveTerm::setMessage("[{$downloaded}/{$total}] Downloading artifacts with concurrency {$this->parallel} ...");
        InteractiveTerm::advance();
    } else {
        $fiber->resume();
    }
}
// all done
if (count($this->artifacts) === 0 && count($fiber_pool) === 0) {
    InteractiveTerm::finish("Downloaded all {$total} artifacts.");
    break;
}

// in Shell->passthru() function:
InteractiveTerm::advance();
\Fiber::suspend();

[crazywhalecc]

[jingjingxyk]

666

[henderkes]

Great! But gmp's website really doesn't like you, hahaha. Maybe we should look for a mirror.

[henderkes]

https://mirrors.dotsrc.org/gnu/gmp/

[jingjingxyk]

https://ftpmirror.gnu.org/gnu/gmp/gmp

[henderkes]

https://ftpmirror.gnu.org/gnu/gmp/gmp

That redirects me to various different sites, all of which return 404 😅