Flow の型チェックを CircleCI から実行する時はワーカーの上限値を設定した方がよかった話

19157 ワード

CircleCI flow JavaScript flowtype JavaScript テキストリンク

前提条件

CircleCI など仮想環境上での実行であること。
flow-bin v0.82.0 以降であること。
型チェックする対象ファイルが多数存在すること。

現象

型チェックの対象となるファイルを大量に置いた状態から Flow の型チェックを実行すると、次のようなエラーメッセージを吐いて停止してしまう。

#!/bin/bash -eo pipefail
npm run flow

> [email protected] flow /home/circleci/repo
> npx flow

Launching Flow server for /home/circleci/repo
Spawned flow server (pid=136)
Logs will go to /tmp/flow/zShomezScirclecizSrepo.log
Monitor logs will go to /tmp/flow/zShomezScirclecizSrepo.monitor_log
Please wait. Server is starting up: \Please wait. Server is initializing (parsed files 3000): -Please wait. Server is initializing (parsed files 10056): \Please wait. Server is initializing (parsed files 10056): |Please wait. Server is initializing (parsed files 10056): /Please wait. Server is initializing (resolving dependencies): -Please wait. Server is initializing (resolving dependencies): \Please wait. Server is initializing (resolving dependencies): |Please wait. Server is initializing (resolving dependencies): /Please wait. Server is initializing (resolving dependencies): -Please wait. Server is initializing (merged files 0/3821 (0.0%)): -Please wait. Server is initializing (merged files 1387/3821 (36.3%)): \Please wait. Server is initializing (merged files 1764/3821 (46.2%)): |Please wait. Server is initializing (merged files 2036/3821 (53.3%)): /Please wait. Server is initializing (merged files 2283/3821 (59.7%)): -Please wait. Server is initializing (merged files 2352/3821 (61.6%)): \Please wait. Server is initializing (merged files 2475/3821 (64.8%)): |Please wait. Server is initializing (merged files 2585/3821 (67.7%)): /Please wait. Server is initializing (merged files 2693/3821 (70.5%)): -Please wait. Server is initializing (merged files 2772/3821 (72.5%)): \Please wait. Server is initializing (merged files 2881/3821 (75.4%)): |Please wait. Server is initializing (merged files 2991/3821 (78.3%)): /Please wait. Server is initializing (merged files 3067/3821 (80.3%)): -Please wait. Server is initializing (merged files 3153/3821 (82.5%)): \The flow server is not responding (3 retries remaining): -Launching Flow server for /home/circleci/repo
Spawned flow server (pid=688)
Logs will go to /tmp/flow/zShomezScirclecizSrepo.log
Monitor logs will go to /tmp/flow/zShomezScirclecizSrepo.monitor_log
Please wait. Server is initializing (parsed files 0): -Please wait. Server is initializing (parsed files 5000): \Please wait. Server is initializing (parsed files 10056): |Please wait. Server is initializing (parsed files 10056): /Please wait. Server is initializing (resolving dependencies): \Please wait. Server is initializing (resolving dependencies): |Please wait. Server is initializing (resolving dependencies): /Please wait. Server is initializing (resolving dependencies): -Please wait. Server is initializing (resolving dependencies): \Please wait. Server is initializing (merged files 0/3821 (0.0%)): \Please wait. Server is initializing (merged files 1407/3821 (36.8%)): |Please wait. Server is initializing (merged files 1791/3821 (46.9%)): /Please wait. Server is initializing (merged files 2097/3821 (54.9%)): -Please wait. Server is initializing (merged files 2236/3821 (58.5%)): \Please wait. Server is initializing (merged files 2388/3821 (62.5%)): |Please wait. Server is initializing (merged files 2428/3821 (63.5%)): /Please wait. Server is initializing (merged files 2577/3821 (67.4%)): -Please wait. Server is initializing (merged files 2668/3821 (69.8%)): \Please wait. Server is initializing (merged files 2734/3821 (71.6%)): |Please wait. Server is initializing (merged files 2837/3821 (74.2%)): /Please wait. Server is initializing (merged files 2914/3821 (76.3%)): -Please wait. Server is initializing (merged files 3027/3821 (79.2%)): \Please wait. Server is initializing (merged files 3170/3821 (83.0%)): |Please wait. Server is initializing (merged files 3287/3821 (86.0%)): /The flow server is not responding (3 retries remaining): \Launching Flow server for /home/circleci/repo
Spawned flow server (pid=1254)

...

Logs will go to /tmp/flow/zShomezScirclecizSrepo.log
Monitor logs will go to /tmp/flow/zShomezScirclecizSrepo.monitor_log
Please wait. Server is initializing (parsed files 0): |Please wait. Server is initializing (parsed files 5000): /Please wait. Server is initializing (parsed files 10056): -Please wait. Server is initializing (parsed files 10056): \Please wait. Server is initializing (parsed files 10056): |Please wait. Server is initializing (resolving dependencies): /Please wait. Server is initializing (resolving dependencies): -Please wait. Server is initializing (resolving dependencies): \Please wait. Server is initializing (resolving dependencies): |Please wait. Server is initializing (waiting for Watchman - giving up in 14 seconds): -Please wait. Server is initializing (merged files 0/3821 (0.0%)): /Please wait. Server is initializing (merged files 1447/3821 (37.9%)): -Please wait. Server is initializing (merged files 1817/3821 (47.6%)): \Please wait. Server is initializing (merged files 2074/3821 (54.3%)): |Please wait. Server is initializing (merged files 2269/3821 (59.4%)): /Please wait. Server is initializing (merged files 2385/3821 (62.4%)): -Please wait. Server is initializing (merged files 2488/3821 (65.1%)): \Please wait. Server is initializing (merged files 2615/3821 (68.4%)): |Please wait. Server is initializing (merged files 2692/3821 (70.5%)): /Please wait. Server is initializing (merged files 2780/3821 (72.8%)): -Please wait. Server is initializing (merged files 2864/3821 (75.0%)): \Please wait. Server is initializing (merged files 2941/3821 (77.0%)): |Please wait. Server is initializing (merged files 3043/3821 (79.6%)): /Lost connection to the flow server (0 retries remaining): \Out of retries, exiting!
npm ERR! code ELIFECYCLE
npm ERR! errno 7
npm ERR! [email protected] flow: `npx flow`
npm ERR! Exit status 7
npm ERR! 
npm ERR! Failed at the [email protected] flow script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2019-02-02T09_12_20_617Z-debug.log
Exited with code 7

一方で手元の環境では問題なく動作する。

ここで Flow のコマンドを flow check に修正し、 CircleCI 上で再度実行すると次のようなエラーメッセージを吐くようになった。

#!/bin/bash -eo pipefail
npm run flow

> [email protected] flow /home/circleci/repo
> npx flow check


Worker interrupted with signal: sigkill
(709) merge_strict_job THROWS: [1] /home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactCurrentFiber.js: Utils_js.Key_not_found("LeaderHeap", "/home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactFiber.js")
Raised at file "hashtbl.ml", line 194, characters 13-28
Called from file "hack/heap/sharedMem.ml", line 1136, characters 22-42


(709) merge_strict_job THROWS: [1] /home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactDebugFiberPerf.js: Utils_js.Key_not_found("LeaderHeap", "/home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactFiber.js")
Raised at file "hashtbl.ml", line 194, characters 13-28
Called from file "hack/heap/sharedMem.ml", line 1136, characters 22-42

...

Worker slave 690 exception: Unix.Unix_error(Unix.EPIPE, "write", "")
Worker slave 690 Potential backtrace:
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 145, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 158, characters 4-63
Called from file "hack/utils/measure.ml", line 238, characters 12-16
Called from file "hack/procs/worker.ml", line 88, characters 14-143
Called from file "hack/procs/worker.ml", line 128, characters 4-37

Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 145, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 158, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
(708) merge_strict_job THROWS: [1] /home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactProfilerTimer.js: Utils_js.Key_not_found("LeaderHeap", "/home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactFiberHostConfig.js")
Raised at file "hashtbl.ml", line 194, characters 13-28
Called from file "hack/heap/sharedMem.ml", line 1136, characters 22-42


Worker slave 708 exception: Unix.Unix_error(Unix.EPIPE, "write", "")
Worker slave 708 Potential backtrace:
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 145, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 158, characters 4-63
Called from file "hack/utils/measure.ml", line 238, characters 12-16
Called from file "hack/procs/worker.ml", line 88, characters 14-143
Called from file "hack/procs/worker.ml", line 128, characters 4-37

npm ERR! code ELIFECYCLE
npm ERR! errno 110
npm ERR! [email protected] flow: `npx flow check`
npm ERR! Exit status 110
npm ERR! 
npm ERR! Failed at the [email protected] flow script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2019-02-02T09_15_56_449Z-debug.log
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 145, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 158, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Exited with code 110

原因

この issue のコメントがわかりやすい。

Flow のオプションである server.max_workers についてはドキュメントに次のような記載がある。

server.max_workers (integer)
The maximum number of workers the Flow server can start. By default, the server will use all available cores.
https://flow.org/en/docs/config/options/#toc-server-max-workers-integer

Flow は型チェックする際にサーバーとワーカーを起動するが、デフォルトで起動可能なワーカーの上限数は CPU のコア数としている。しかし仮想環境化では実際に利用可能な CPU のコア数よりも多いワーカーを起動可能となってしまうため、メモリを食い潰してプロセスが強制終了されてしまうケースがあるようだ。

対策

issue 付いたコメントのように server.max_workers をパフォーマンス上許容できる値に指定すると良い。私の環境では 1 だと時間が掛かり過ぎるので 4 にした。

[options]
server.max_workers=4

Author And Source

この問題について(Flow の型チェックを CircleCI から実行する時はワーカーの上限値を設定した方がよかった話), 我々は、より多くの情報をここで見つけました https://qiita.com/sakai-akinobu/items/527dc606adc208a729c2

著者帰属：元の著者の情報は、元のURLに含まれています。著作権は原作者に属する。

Content is automatically searched and collected through network algorithms . If there is a violation . Please contact us . We will adjust (correct author information ,or delete content ) as soon as possible .

pyinstallerのインストール使用中に発生したエラー

python爬虫類(二):爬虫効率の向上