public inbox for goredo-devel@lists.cypherpunks.ru Atom feed
* Support for virtual targets @ 2022-04-16 19:13 Jan Niklas Böhm 2022-04-16 21:11 ` Sergey Matveev 0 siblings, 1 reply; 6+ messages in thread From: Jan Niklas Böhm @ 2022-04-16 19:13 UTC (permalink / raw) To: goredo-devel Hello, I was wondering why this implementations lists under its notes [1] (third point) that: - empty targets are considered always out of date This caught me a bit by surprise and I was wondering whether its possible to actually check whether the target is ood by looking at its dependencies, similar to how redo-python does it. The use case is also given in those documents, e. g. compiling a latex document [2]. From what I was able to gather the corresponding lines are probably l. 130 and/or l.158 in ood.go. I am going off of the most recent version 1.24.0. If I can help with this, please let me know, although I am afraid that I have not written anything in Go so far. Cheers Nik [1]: http://www.goredo.cypherpunks.ru/Notes.html [2]: https://redo.readthedocs.io/en/latest/cookbook/latex/#virtual-targets-side-effects-and-multiple-outputs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Support for virtual targets 2022-04-16 19:13 Support for virtual targets Jan Niklas Böhm @ 2022-04-16 21:11 ` Sergey Matveev 2022-04-17 20:37 ` Resolving paths within symlinked dirs (Re: Support for virtual targets) Jan Niklas Böhm 0 siblings, 1 reply; 6+ messages in thread From: Sergey Matveev @ 2022-04-16 21:11 UTC (permalink / raw) To: goredo-devel [-- Attachment #1: Type: text/plain, Size: 2803 bytes --] Greetings! *** Jan Niklas Böhm [2022-04-16 21:13]: >This caught me a bit by surprise and I was wondering whether its possible to >actually check whether the target is ood by looking at its dependencies, >similar to how redo-python does it. The use case is also given in those >documents, e. g. compiling a latex document [2]. I am strongly against checking of dependencies for those "virtual" (as apenwarr/redo calls them) targets. This is a convenient feature. apenwarr/redo's LaTeX-related example shows how side-effects can be conveniently used for producing multiple files. That is useful thing, that I personally use with LaTeX and Texinfo documentation building too. But there will be no difference in behaviour if that virtual target produces any output or not. You can add "touch $3" to its end and everything will work and behave the same way in that apenwarr/redo's example. apenwarr/redo mentions that those virtual targets are called .PHONY ones in make-world. And my make's man page says indeed: .PHONY The target does not correspond to an actual file; it is always considered to be out of date, and will not be created with the -t option. So even in make-world, .PHONY by definition is something always out-of-date. But apenwarr/redo's virtual target does honest OOD checking and it won't be *always* out-of-date and can not be claimed to be .PHONE-analogue. apenwarr/redo just has not ability to create real phony targets, except for using an additional workaround command like redo-always, that is not supported in all redo implementations and its usage won't be portable. In comparison, goredo's non-existent targets (now I slightly corrected documentation and replaced some "empty" with "non-existent", because it was confusing: "empty" is just something having zero size) are the real .PHONY ones -- they are always OOD. If you want to make a target that honestly does OOD checking, then explicitly create a file ("touch $3" at least). So in goredo you can create both real "phony" targets (without producing any output) and "virtual" targets (by making an empty file). apenwarr/redo lacks "phony" targets possibility (except for additional redo-always invocation). Phony targets are used mainly for targets like "all"/"install" in practice, so as a rule they are called manually with "redo" command, not redo-ifchange, and that is why they are forcefully run anyway, like an ordinary phony-target should do. "Virtual" targets, with explicit "touch $3" are also portable across apenwarr/redo, redo-c and similar implementations -- so hardly there are any portability issues. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Resolving paths within symlinked dirs (Re: Support for virtual targets) 2022-04-16 21:11 ` Sergey Matveev @ 2022-04-17 20:37 ` Jan Niklas Böhm 2022-04-22 9:27 ` Resolving paths within symlinked dirs targets) Sergey Matveev 0 siblings, 1 reply; 6+ messages in thread From: Jan Niklas Böhm @ 2022-04-17 20:37 UTC (permalink / raw) To: goredo-devel Thanks for the answer, that does make things much clearer! I will then change my approach in order to create a file instead of relying on the apenwarr/redo-specific behavior. Thanks for the detailed explanation! > So in goredo you can create both real "phony" targets (without producing > any output) and "virtual" targets (by making an empty file). > apenwarr/redo lacks "phony" targets possibility (except for additional > redo-always invocation). Phony targets are used mainly for targets like > "all"/"install" in practice, so as a rule they are called manually with > "redo" command, not redo-ifchange, and that is why they are forcefully > run anyway, like an ordinary phony-target should do. "Virtual" targets, > with explicit "touch $3" are also portable across apenwarr/redo, redo-c > and similar implementations -- so hardly there are any portability issues. Now I do agree that this is the more sensible behavior of the program, I was not aware that this is inherited from how Makefiles operate. This actually send me down a rabbit hole as the target was still perpetually ood despite creating an output file in the former virtual/phony do file. I think I have chased down the problem after investigating this for a good part of today (and in hindsight it is probably trivial). The issue is that source files are not always correctly resolved when redo-ing files in symlinked directories. I hope the following MWE illustrates the problem. $ mkdir -p three/folders/deep $ ln -s three/folders/deep symlink $ export DIR=`pwd` $ mkdir elsewhere $ echo foo > elsewhere/srcfile $ cat << EOF > three/folders/deep/target.do redo-ifchange $DIR/elsewhere/srcfile cat $DIR/elsewhere/srcfile EOF When going into the directory “three/folders/deep”, and typing “redo target” everything works as expected. But when cd-ing into the folder “symlink”, the target will always be considered out of date; since it is non-existent. From what I understand the issue is that the absolute path is converted to a relative path, which is relative to the proper directory. On the other hand, the symlink then has a different nesting depth and resolving the relative path from the symlinked directory will point to a non-existent file; hence the target being ood. This problem is not related to the initial mail and obviously virtual targets would not help here. I am not entirely sure on what would be the best approach in order to correct this behavior, maybe the cwd can be resolved through the file system prior to checking for ood targets. I hope the example is illustrative, I can try and expand on the issue, should that be necessary. Thanks for the quick response and nice explanation in the first place! ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Resolving paths within symlinked dirs targets) 2022-04-17 20:37 ` Resolving paths within symlinked dirs (Re: Support for virtual targets) Jan Niklas Böhm @ 2022-04-22 9:27 ` Sergey Matveev 2022-04-22 20:42 ` Resolving paths within symlinked dirs targets Jan Niklas Böhm 0 siblings, 1 reply; 6+ messages in thread From: Sergey Matveev @ 2022-04-22 9:27 UTC (permalink / raw) To: goredo-devel [-- Attachment #1: Type: text/plain, Size: 2363 bytes --] Greetings! *** Jan Niklas Böhm [2022-04-17 22:37]: >From what I understand the issue is that the absolute path is converted to a >relative path, which is relative to the proper directory. On the other >hand, the symlink then has a different nesting depth and resolving the >relative path from the symlinked directory will point to a non-existent >file; hence the target being ood. Yes, everything is right. >I am not entirely sure on what would be the >best approach in order to correct this behavior, maybe the cwd can be >resolved through the file system prior to checking for ood targets. Actually initially goredo used to resolve its "physical" (pwd -P) path to the current directory. But one of its users gave very good example where it makes things very inconvenient in practice: for example in FreeBSD /home/user is actually a symlink to /usr/home/user by default. That is why pwd -P will return paths like that, when located in /home/user: ../../../../../../home/user/docs/3rd/goredo/all.do ../../../../../../home/user/docs/3rd/goredo/default.do ../../../../../../home/user/docs/3rd/default.do ../../../../../../home/user/docs/default.do ../../../../../../home/user/default.do ../../../../../../home/default.do ../../../../../../default.do and that is clearly something user do not expect and do not want to. So the better way from the point of convenience was to behave as pwd -L by default. Probably we can check both -L/-P behaviours during path resolution to the target to check if the target is actually something known and existing? Unfortunately no, we can not do that at all, according to: https://unix.stackexchange.com/questions/79571/symbolic-link-recursion-what-makes-it-reset/79621#79621 With the path a/b/c, where a/b/c is a symlink to /somewhere, a/b/c/../d may lead both to a/b/d, or to /somewhere/d, that are completely different paths, but pretty valid from the point of symlinked paths. And both a/b/d and /somewhere/d can be existing files. So there just can not be any "valid" way of behaving when dealing with symlinks. That is one of the reason Plan9 completely criticise and abandoned that idea: http://harmful.cat-v.org/software/symlinks -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Resolving paths within symlinked dirs targets 2022-04-22 9:27 ` Resolving paths within symlinked dirs targets) Sergey Matveev @ 2022-04-22 20:42 ` Jan Niklas Böhm 2022-04-24 18:34 ` Sergey Matveev 0 siblings, 1 reply; 6+ messages in thread From: Jan Niklas Böhm @ 2022-04-22 20:42 UTC (permalink / raw) To: goredo-devel Thanks for the thorough response! I appreciate it. > Actually initially goredo used to resolve its "physical" (pwd -P) path > to the current directory. But one of its users gave very good example > where it makes things very inconvenient in practice: for example in > FreeBSD /home/user is actually a symlink to /usr/home/user by default. > That is why pwd -P will return paths like that, when located in /home/user: > > ../../../../../../home/user/docs/3rd/goredo/all.do > ../../../../../../home/user/docs/3rd/goredo/default.do > ../../../../../../home/user/docs/3rd/default.do > ../../../../../../home/user/docs/default.do > ../../../../../../home/user/default.do > ../../../../../../home/default.do > ../../../../../../default.do > > and that is clearly something user do not expect and do not want to. So > the better way from the point of convenience was to behave as pwd -L by > default. I can see why this is not ideal from a users perspective, but to me it strikes me as the correct location of the path. Having those paths fully resolved as well should still point to the correct location, as the /home/user path should resolve properly to its --physical location, or am I mistaken? The above example could be completely resolved by resolving both the pwd and the target (in this case the .do files) to their physical location. That should turn your example into (assuming that pwd = /home/user/): docs/3rd/goredo/all.do docs/3rd/goredo/default.do docs/3rd/default.do docs/default.do default.do ../default.do ../../default.do This should be equal to the current behavior, but irrespective of whether we have cd'd into the directory through their physical path or the symlink, unless I am missing something. Of course this will come at a minor performance penalty since all the paths and the pwd have to be resolved through the file system, but I think that it is worth it. > Probably we can check both -L/-P behaviours during path resolution to > the target to check if the target is actually something known and > existing? Unfortunately no, we can not do that at all, according to: > https://unix.stackexchange.com/questions/79571/symbolic-link-recursion-what-makes-it-reset/79621#79621 > With the path a/b/c, where a/b/c is a symlink to /somewhere, a/b/c/../d > may lead both to a/b/d, or to /somewhere/d, that are completely > different paths, but pretty valid from the point of symlinked paths. And > both a/b/d and /somewhere/d can be existing files. So there just can not > be any "valid" way of behaving when dealing with symlinks. That is one > of the reason Plan9 completely criticise and abandoned that idea: > http://harmful.cat-v.org/software/symlinks I agree that this option cannot work out and is not worth pursuing. Thank you again, I really appreciate your answers. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Resolving paths within symlinked dirs targets 2022-04-22 20:42 ` Resolving paths within symlinked dirs targets Jan Niklas Böhm @ 2022-04-24 18:34 ` Sergey Matveev 0 siblings, 0 replies; 6+ messages in thread From: Sergey Matveev @ 2022-04-24 18:34 UTC (permalink / raw) To: goredo-devel [-- Attachment #1: Type: text/plain, Size: 811 bytes --] *** Jan Niklas Böhm [2022-04-22 22:42]: >The above example could be completely resolved by resolving both the pwd and >the target (in this case the .do files) to their physical location. That >should turn your example into (assuming that pwd = /home/user/): Currently I can not analyze that deeply, but probably that will work. I tried to modify goredo locally and resolve physical paths to all targets, but either I made a mistake somewhere (my implementation uses filepath.EvalSymlinks) and various tests are failing, or there is some issue with that idea. Currently just do not have enough time, but I hope to give detailed answer (or release, if there are no issues) the next week. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-04-24 18:35 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-16 19:13 Support for virtual targets Jan Niklas Böhm 2022-04-16 21:11 ` Sergey Matveev 2022-04-17 20:37 ` Resolving paths within symlinked dirs (Re: Support for virtual targets) Jan Niklas Böhm 2022-04-22 9:27 ` Resolving paths within symlinked dirs targets) Sergey Matveev 2022-04-22 20:42 ` Resolving paths within symlinked dirs targets Jan Niklas Böhm 2022-04-24 18:34 ` Sergey Matveev