public inbox for goredo-devel@lists.cypherpunks.ru
Atom feed
From: spacefrogg <spacefrogg-goredo@spacefrogg•net>
To: Goredo Devel <goredo-devel@lists.cypherpunks.ru>
Subject: Dependency collection takes very long
Date: Thu, 28 Sep 2023 12:05:08 +0000 [thread overview]
Message-ID: <bd1ec0b6419ef2f3ebd5c6cca78da433@spacefrogg.net> (raw)
Hi!
I have a not-so-small project. The final target has about 5700
dependencies boiling down to 1300 unique dependencies, both numbers
include transient dependencies. redo-targets reports about 270 targets,
so the rest are source files (≈1050).
The first build runs very smooth (2min single job, 50secs with 11 jobs).
Starting with the second build, though, goredo takes tens of seconds of
busy CPU work after logging "dbg collecting deps" in the debug output.
`iotop` shows that it is busy doing I/O.
N.B. `time` records about 2m50s real time while redo only counts 1m23s.
So, some significant code executes before redo starts counting the time.
As a comparison, apenwarr/redo needs a fraction of a second to check all
dependencies. (It does no source file hashing, though.)
redo-targets returns instantly, but redo-sources gets stuck as well,
thrashing the disk at a rate of 12MB/s taking almost 11m to finish. This
means it touched almost 8GB! So, I assume that something is off when
redo encounters a source file.
I did not change the default REDO_INODE_TRUST. So, I assume redo is
using ctime checking. My ctimes are reliable. I tested with goredo 1.30
and 1.32.
By the looks of it, redo always hashes source files. But even if, this
takes way too much time for the actual amount of data (≈120MB).
–Michael
next reply other threads:[~2023-09-28 12:05 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-28 12:05 spacefrogg [this message]
2023-09-30 16:47 ` Dependency collection takes very long Sergey Matveev
2023-09-30 21:02 ` Sergey Matveev
2023-10-01 11:53 ` goredo
2023-10-02 11:01 ` Sergey Matveev
2023-10-07 15:29 ` Sergey Matveev