public inbox for goredo-devel@lists.cypherpunks.ru
Atom feed
* Support for virtual targets
@ 2022-04-16 19:13 Jan Niklas Böhm
2022-04-16 21:11 ` Sergey Matveev
0 siblings, 1 reply; 6+ messages in thread
From: Jan Niklas Böhm @ 2022-04-16 19:13 UTC (permalink / raw)
To: goredo-devel
Hello,
I was wondering why this implementations lists under its notes [1]
(third point) that:
- empty targets are considered always out of date
This caught me a bit by surprise and I was wondering whether its
possible to actually check whether the target is ood by looking at its
dependencies, similar to how redo-python does it. The use case is also
given in those documents, e. g. compiling a latex document [2].
From what I was able to gather the corresponding lines are probably l.
130 and/or l.158 in ood.go. I am going off of the most recent version
1.24.0. If I can help with this, please let me know, although I am
afraid that I have not written anything in Go so far.
Cheers
Nik
[1]: http://www.goredo.cypherpunks.ru/Notes.html
[2]:
https://redo.readthedocs.io/en/latest/cookbook/latex/#virtual-targets-side-effects-and-multiple-outputs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Support for virtual targets
2022-04-16 19:13 Support for virtual targets Jan Niklas Böhm
@ 2022-04-16 21:11 ` Sergey Matveev
2022-04-17 20:37 ` Resolving paths within symlinked dirs (Re: Support for virtual targets) Jan Niklas Böhm
0 siblings, 1 reply; 6+ messages in thread
From: Sergey Matveev @ 2022-04-16 21:11 UTC (permalink / raw)
To: goredo-devel
[-- Attachment #1: Type: text/plain, Size: 2803 bytes --]
Greetings!
*** Jan Niklas Böhm [2022-04-16 21:13]:
>This caught me a bit by surprise and I was wondering whether its possible to
>actually check whether the target is ood by looking at its dependencies,
>similar to how redo-python does it. The use case is also given in those
>documents, e. g. compiling a latex document [2].
I am strongly against checking of dependencies for those "virtual" (as
apenwarr/redo calls them) targets. This is a convenient feature.
apenwarr/redo's LaTeX-related example shows how side-effects can be
conveniently used for producing multiple files. That is useful thing,
that I personally use with LaTeX and Texinfo documentation building too.
But there will be no difference in behaviour if that virtual target
produces any output or not. You can add "touch $3" to its end and
everything will work and behave the same way in that apenwarr/redo's
example.
apenwarr/redo mentions that those virtual targets are called .PHONY ones
in make-world. And my make's man page says indeed:
.PHONY The target does not correspond to an actual file; it is always
considered to be out of date, and will not be created with the
-t option.
So even in make-world, .PHONY by definition is something always
out-of-date. But apenwarr/redo's virtual target does honest OOD
checking and it won't be *always* out-of-date and can not be claimed to
be .PHONE-analogue. apenwarr/redo just has not ability to create real
phony targets, except for using an additional workaround command like
redo-always, that is not supported in all redo implementations and its
usage won't be portable.
In comparison, goredo's non-existent targets (now I slightly corrected
documentation and replaced some "empty" with "non-existent", because it
was confusing: "empty" is just something having zero size) are the real
.PHONY ones -- they are always OOD. If you want to make a target that
honestly does OOD checking, then explicitly create a file ("touch $3" at
least).
So in goredo you can create both real "phony" targets (without producing
any output) and "virtual" targets (by making an empty file).
apenwarr/redo lacks "phony" targets possibility (except for additional
redo-always invocation). Phony targets are used mainly for targets like
"all"/"install" in practice, so as a rule they are called manually with
"redo" command, not redo-ifchange, and that is why they are forcefully
run anyway, like an ordinary phony-target should do. "Virtual" targets,
with explicit "touch $3" are also portable across apenwarr/redo, redo-c
and similar implementations -- so hardly there are any portability issues.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Resolving paths within symlinked dirs (Re: Support for virtual targets)
2022-04-16 21:11 ` Sergey Matveev
@ 2022-04-17 20:37 ` Jan Niklas Böhm
2022-04-22 9:27 ` Resolving paths within symlinked dirs targets) Sergey Matveev
0 siblings, 1 reply; 6+ messages in thread
From: Jan Niklas Böhm @ 2022-04-17 20:37 UTC (permalink / raw)
To: goredo-devel
Thanks for the answer, that does make things much clearer! I will then
change my approach in order to create a file instead of relying on the
apenwarr/redo-specific behavior. Thanks for the detailed explanation!
> So in goredo you can create both real "phony" targets (without producing
> any output) and "virtual" targets (by making an empty file).
> apenwarr/redo lacks "phony" targets possibility (except for additional
> redo-always invocation). Phony targets are used mainly for targets like
> "all"/"install" in practice, so as a rule they are called manually with
> "redo" command, not redo-ifchange, and that is why they are forcefully
> run anyway, like an ordinary phony-target should do. "Virtual" targets,
> with explicit "touch $3" are also portable across apenwarr/redo, redo-c
> and similar implementations -- so hardly there are any portability issues.
Now I do agree that this is the more sensible behavior of the program, I
was not aware that this is inherited from how Makefiles operate.
This actually send me down a rabbit hole as the target was still
perpetually ood despite creating an output file in the former
virtual/phony do file. I think I have chased down the problem after
investigating this for a good part of today (and in hindsight it is
probably trivial). The issue is that source files are not always
correctly resolved when redo-ing files in symlinked directories. I hope
the following MWE illustrates the problem.
$ mkdir -p three/folders/deep
$ ln -s three/folders/deep symlink
$ export DIR=`pwd`
$ mkdir elsewhere
$ echo foo > elsewhere/srcfile
$ cat << EOF > three/folders/deep/target.do
redo-ifchange $DIR/elsewhere/srcfile
cat $DIR/elsewhere/srcfile
EOF
When going into the directory “three/folders/deep”, and typing “redo
target” everything works as expected. But when cd-ing into the folder
“symlink”, the target will always be considered out of date; since it is
non-existent.
From what I understand the issue is that the absolute path is converted
to a relative path, which is relative to the proper directory. On the
other hand, the symlink then has a different nesting depth and resolving
the relative path from the symlinked directory will point to a
non-existent file; hence the target being ood.
This problem is not related to the initial mail and obviously virtual
targets would not help here. I am not entirely sure on what would be
the best approach in order to correct this behavior, maybe the cwd can
be resolved through the file system prior to checking for ood targets.
I hope the example is illustrative, I can try and expand on the issue,
should that be necessary. Thanks for the quick response and nice
explanation in the first place!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Resolving paths within symlinked dirs targets)
2022-04-17 20:37 ` Resolving paths within symlinked dirs (Re: Support for virtual targets) Jan Niklas Böhm
@ 2022-04-22 9:27 ` Sergey Matveev
2022-04-22 20:42 ` Resolving paths within symlinked dirs targets Jan Niklas Böhm
0 siblings, 1 reply; 6+ messages in thread
From: Sergey Matveev @ 2022-04-22 9:27 UTC (permalink / raw)
To: goredo-devel
[-- Attachment #1: Type: text/plain, Size: 2363 bytes --]
Greetings!
*** Jan Niklas Böhm [2022-04-17 22:37]:
>From what I understand the issue is that the absolute path is converted to a
>relative path, which is relative to the proper directory. On the other
>hand, the symlink then has a different nesting depth and resolving the
>relative path from the symlinked directory will point to a non-existent
>file; hence the target being ood.
Yes, everything is right.
>I am not entirely sure on what would be the
>best approach in order to correct this behavior, maybe the cwd can be
>resolved through the file system prior to checking for ood targets.
Actually initially goredo used to resolve its "physical" (pwd -P) path
to the current directory. But one of its users gave very good example
where it makes things very inconvenient in practice: for example in
FreeBSD /home/user is actually a symlink to /usr/home/user by default.
That is why pwd -P will return paths like that, when located in /home/user:
../../../../../../home/user/docs/3rd/goredo/all.do
../../../../../../home/user/docs/3rd/goredo/default.do
../../../../../../home/user/docs/3rd/default.do
../../../../../../home/user/docs/default.do
../../../../../../home/user/default.do
../../../../../../home/default.do
../../../../../../default.do
and that is clearly something user do not expect and do not want to. So
the better way from the point of convenience was to behave as pwd -L by
default.
Probably we can check both -L/-P behaviours during path resolution to
the target to check if the target is actually something known and
existing? Unfortunately no, we can not do that at all, according to:
https://unix.stackexchange.com/questions/79571/symbolic-link-recursion-what-makes-it-reset/79621#79621
With the path a/b/c, where a/b/c is a symlink to /somewhere, a/b/c/../d
may lead both to a/b/d, or to /somewhere/d, that are completely
different paths, but pretty valid from the point of symlinked paths. And
both a/b/d and /somewhere/d can be existing files. So there just can not
be any "valid" way of behaving when dealing with symlinks. That is one
of the reason Plan9 completely criticise and abandoned that idea:
http://harmful.cat-v.org/software/symlinks
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Resolving paths within symlinked dirs targets
2022-04-22 9:27 ` Resolving paths within symlinked dirs targets) Sergey Matveev
@ 2022-04-22 20:42 ` Jan Niklas Böhm
2022-04-24 18:34 ` Sergey Matveev
0 siblings, 1 reply; 6+ messages in thread
From: Jan Niklas Böhm @ 2022-04-22 20:42 UTC (permalink / raw)
To: goredo-devel
Thanks for the thorough response! I appreciate it.
> Actually initially goredo used to resolve its "physical" (pwd -P) path
> to the current directory. But one of its users gave very good example
> where it makes things very inconvenient in practice: for example in
> FreeBSD /home/user is actually a symlink to /usr/home/user by default.
> That is why pwd -P will return paths like that, when located in /home/user:
>
> ../../../../../../home/user/docs/3rd/goredo/all.do
> ../../../../../../home/user/docs/3rd/goredo/default.do
> ../../../../../../home/user/docs/3rd/default.do
> ../../../../../../home/user/docs/default.do
> ../../../../../../home/user/default.do
> ../../../../../../home/default.do
> ../../../../../../default.do
>
> and that is clearly something user do not expect and do not want to. So
> the better way from the point of convenience was to behave as pwd -L by
> default.
I can see why this is not ideal from a users perspective, but to me it
strikes me as the correct location of the path. Having those paths
fully resolved as well should still point to the correct location, as
the /home/user path should resolve properly to its --physical location,
or am I mistaken?
The above example could be completely resolved by resolving both the pwd
and the target (in this case the .do files) to their physical location.
That should turn your example into (assuming that pwd = /home/user/):
docs/3rd/goredo/all.do
docs/3rd/goredo/default.do
docs/3rd/default.do
docs/default.do
default.do
../default.do
../../default.do
This should be equal to the current behavior, but irrespective of
whether we have cd'd into the directory through their physical path or
the symlink, unless I am missing something.
Of course this will come at a minor performance penalty since all the
paths and the pwd have to be resolved through the file system, but I
think that it is worth it.
> Probably we can check both -L/-P behaviours during path resolution to
> the target to check if the target is actually something known and
> existing? Unfortunately no, we can not do that at all, according to:
> https://unix.stackexchange.com/questions/79571/symbolic-link-recursion-what-makes-it-reset/79621#79621
> With the path a/b/c, where a/b/c is a symlink to /somewhere, a/b/c/../d
> may lead both to a/b/d, or to /somewhere/d, that are completely
> different paths, but pretty valid from the point of symlinked paths. And
> both a/b/d and /somewhere/d can be existing files. So there just can not
> be any "valid" way of behaving when dealing with symlinks. That is one
> of the reason Plan9 completely criticise and abandoned that idea:
> http://harmful.cat-v.org/software/symlinks
I agree that this option cannot work out and is not worth pursuing.
Thank you again, I really appreciate your answers.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Resolving paths within symlinked dirs targets
2022-04-22 20:42 ` Resolving paths within symlinked dirs targets Jan Niklas Böhm
@ 2022-04-24 18:34 ` Sergey Matveev
0 siblings, 0 replies; 6+ messages in thread
From: Sergey Matveev @ 2022-04-24 18:34 UTC (permalink / raw)
To: goredo-devel
[-- Attachment #1: Type: text/plain, Size: 811 bytes --]
*** Jan Niklas Böhm [2022-04-22 22:42]:
>The above example could be completely resolved by resolving both the pwd and
>the target (in this case the .do files) to their physical location. That
>should turn your example into (assuming that pwd = /home/user/):
Currently I can not analyze that deeply, but probably that will work. I
tried to modify goredo locally and resolve physical paths to all
targets, but either I made a mistake somewhere (my implementation uses
filepath.EvalSymlinks) and various tests are failing, or there is some
issue with that idea. Currently just do not have enough time, but I hope
to give detailed answer (or release, if there are no issues) the next week.
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-04-24 18:35 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-16 19:13 Support for virtual targets Jan Niklas Böhm
2022-04-16 21:11 ` Sergey Matveev
2022-04-17 20:37 ` Resolving paths within symlinked dirs (Re: Support for virtual targets) Jan Niklas Böhm
2022-04-22 9:27 ` Resolving paths within symlinked dirs targets) Sergey Matveev
2022-04-22 20:42 ` Resolving paths within symlinked dirs targets Jan Niklas Böhm
2022-04-24 18:34 ` Sergey Matveev