public inbox for nncp-devel@lists.cypherpunks.ru Atom feed
* NNCP in Debian/Ubuntu Update @ 2021-09-20 13:05 John Goerzen 2021-09-20 19:02 ` Sergey Matveev 0 siblings, 1 reply; 6+ messages in thread From: John Goerzen @ 2021-09-20 13:05 UTC (permalink / raw) To: nncp-devel Hello everyone, At long last, NNCP has been accepted into Debian unstable. This is great news. It means NNCP will be in future Debian (and derivatives such as Ubuntu) releases. From here, I hope to get it migrated to testing in the next couple of weeks. Once in testing, I can build it for buster and bullseye backports. This will make it apt-get installable on lots of current machines, including Raspberry Pis and anyone running the current or prior release of Debian. - John ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP in Debian/Ubuntu Update 2021-09-20 13:05 NNCP in Debian/Ubuntu Update John Goerzen @ 2021-09-20 19:02 ` Sergey Matveev 2021-09-20 20:10 ` John Goerzen 0 siblings, 1 reply; 6+ messages in thread From: Sergey Matveev @ 2021-09-20 19:02 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 769 bytes --] Greetings! *** John Goerzen [2021-09-20 08:05]: >At long last, NNCP has been accepted into Debian unstable. Wonderful! Thank you! Just for the interest I looked at https://salsa.debian.org/go-team/packages/nncp.git I have never dealt with Debian packages (well, 15+ years ago played only with the one), so just my misunderstanding, but it seems that NNCP is not built from the tarball? I see some dependencies on external Go-related packages and documentation building in debian/rules file. I remind that NNCP's tarballs contain all Go dependencies, that are statically linked in anyway. And they contain both Info and HTML documentation. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP in Debian/Ubuntu Update 2021-09-20 19:02 ` Sergey Matveev @ 2021-09-20 20:10 ` John Goerzen 2021-09-20 20:30 ` Sergey Matveev 0 siblings, 1 reply; 6+ messages in thread From: John Goerzen @ 2021-09-20 20:10 UTC (permalink / raw) To: Sergey Matveev; +Cc: nncp-devel On Mon, Sep 20 2021, Sergey Matveev wrote: > Greetings! > > *** John Goerzen [2021-09-20 08:05]: >>At long last, NNCP has been accepted into Debian unstable. > > Wonderful! Thank you! > > Just for the interest I looked at > https://salsa.debian.org/go-team/packages/nncp.git > I have never dealt with Debian packages (well, 15+ years ago > played only > with the one), so just my misunderstanding, but it seems that > NNCP is > not built from the tarball? I see some dependencies on external > Go-related packages and documentation building in debian/rules > file. I > remind that NNCP's tarballs contain all Go dependencies, that > are > statically linked in anyway. And they contain both Info and HTML > documentation. Yes. Actually this is the reason it took me so long to get it in Debian; I had to learn about Go building and stuff. So Debian has a very long-standing policy, going back decades, of not building dependencies from in-tree. Even back in the C days, a lot of packages would require other libraries and include them in their tarballs. That caused a number of problems at the distribution scale. For instance, if there's a new OpenSSL vuln and various packages build from OpenSSL in-tarball, now instead of patching one thing, you first have to figure out where all it's embedded (a challenge itself) and then patch and rebuild all of those. The same goes for bugs in OpenSSL, etc. It's not particularly sustainable. There is also another long-standing policy that every Debian package must be buildable using only what is in Debian, and without network access. This is enforced at the CI level. The rationale for this includes several aspects: 1) A build that downloads things from the Internet can break if those things go away; 2) it makes it easier to inject malware into the system; 3) it makes it harder to audit for license compliance; 4) it can lead to unintentional violations of the Debian Free Software Guidelines (if sources are downloaded but not distributed, this can also be a GPL violation). So the Go and Rust ecosystems (plus others like Python) don't use the go/cargo/pip build mechanism to just download stuff off the Internet; they require that the dependencies already exist in Debian, and build using only those. I did initially try to just use the tarball, but that was (somewhat predictably) rejected for these reasons. So, I made separate packages for all the things that weren't already in Debian (blake3, balloon, etc) and got those into the archive. Then NNCP could go in. https://go-team.pages.debian.net/packaging.html talks about this. The tarball is absolutely the right thing for users. I like what you've done there. It just wasn't the most suitable for a distribution (or, shall we say, a distribution with Debian's policies). There is automated tooling to bring go stuff into Debian, since this is far from the only such case. It is centered around the go convention of using git. It mostly worked, here. I had to patch up a few things but generally it worked well. In fact, I do have a 7.7.0 release prepared -- the upgrade was one command -- which I will upload probably tomorrow once all of its dependencies have hit the mirrors. The uilive thing is a bit of an odd one; it was still in-tree in the git repo and it survived review so I guess it works for now :-) I also had to do a bit of hacking around the source tree; I don't quite understand it, but there was something relating to nncp/v7 that the automated tooling didn't quite deal well with (you can see this in the debian/rules file). So anyway, the long and short of it is, it is done, in the Debian way, and is working and will move ahead! - John ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP in Debian/Ubuntu Update 2021-09-20 20:10 ` John Goerzen @ 2021-09-20 20:30 ` Sergey Matveev 2021-09-20 21:13 ` John Goerzen 0 siblings, 1 reply; 6+ messages in thread From: Sergey Matveev @ 2021-09-20 20:30 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 2187 bytes --] *** John Goerzen [2021-09-20 15:10]: >So Debian has a very long-standing policy, going back decades, of not >building dependencies from in-tree. Hmm... I probably can understand that for dynamically linked executables, but that seems strange for statically linked ones, that Go does by default. Moreover, Go's go.mod/go.sum are tied not to some abstract version of the library, but literally to its exact commit/tag. And even in one Go program you can easily use different versions of the same library, that will force you to install golang-whatever-* of various versions. All of that seems very strange to me, but it is Debian's decision, ok :-) >There is also another long-standing policy that every Debian package must be >buildable using only what is in Debian, and without network access. That is good, without any doubts! >It just wasn't the most suitable for a distribution (or, shall >we say, a distribution with Debian's policies). And that frightens me, because if someone want to report a bug, then it have to report about dozen of dependencies and their exact versions, instead of reporting only of Go version and NNCP's one. Oh well... personally I slowly moving more and more to GNU Stow type of packages management, manually building software :-) >The uilive thing is a bit of an odd one; it was still in-tree in the git >repo and it survived review so I guess it works for now :-) As I remember I copied it and modified. It is not original version. Do not remember why, but I believe to strip off various complicated unneeded stuff. >I also had to do a bit of hacking around the source tree; I don't quite >understand it, but there was something relating to nncp/v7 that the >automated tooling didn't quite deal well with (you can see this in the >debian/rules file). Strange, have no ideas where "src/go.cypherpunks.ru/nncp/v7/src" path comes from. But yeah, last "src/" is not needed. >So anyway, the long and short of it is, it is done, in the Debian way, and >is working and will move ahead! Cool! -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP in Debian/Ubuntu Update 2021-09-20 20:30 ` Sergey Matveev @ 2021-09-20 21:13 ` John Goerzen 2021-09-21 10:10 ` Sergey Matveev 0 siblings, 1 reply; 6+ messages in thread From: John Goerzen @ 2021-09-20 21:13 UTC (permalink / raw) To: Sergey Matveev; +Cc: nncp-devel On Mon, Sep 20 2021, Sergey Matveev wrote: > *** John Goerzen [2021-09-20 15:10]: >>So Debian has a very long-standing policy, going back decades, >>of not >>building dependencies from in-tree. > > Hmm... I probably can understand that for dynamically linked > executables, but that seems strange for statically linked ones, > that Go does by default. Moreover, Go's go.mod/go.sum are tied > not to some abstract version of the library, but literally to > its > exact commit/tag. And even in one Go program you can easily use > different versions of the same library, that will force you to > install golang-whatever-* of various versions. All of that seems > very strange to me, but it is Debian's decision, ok :-) Fundamentally I can't just change that on my own, so our conversation here is discussing something that "is what it is", regardless of whether we agree with it. But, hey, I like conversation :-) I'm not as familiar with the Go build, but if I could express this in terms of rust: In Rust, there is a Cargo.toml, which defines the general requirements for dependencies (ie, package X, version 0.2.*) and then a cargo.lock which records the precise version and hash of the dependency used for a canonical build. There are pros and cons to this... The pro is, of course, that you can easily get a theoretically identical build from anywhere. And the con is that you may have dozens of different versions of the same package in use on a system, making it difficult to patch bugs and security issues. So any person can come down on one side or the other of that, and I totally understand that. Debian as a project has chosen where it comes down, and carries the pros and cons with it. FreeBSD and Arch probably come down differently, and live with the pros and cons of their choices too. One thing that flows out of these decisions is the possibility of unattended-upgrades existing in Debian, but really not so much in FreeBSD ports or Arch. unattended-upgrades basically can patch your system for security bugs automatically from cron. That's not for everyone, but I love it. It's possible primarily because Debian backports security fixes to the version in stable. That is, you get a narrowly-targeted patch that has a low risk of introducing issues because it isn't pulling in major new versions of software and such. But the choice here also helps, because if the distro follows cargo.lock of the go equivalent, that can preclude the possibility of narrowly-targeted patches and also expand the work of the security team to unsustainable levels. Tradeoffs abound. What's right for Debian isn't right for everybody. Debian prizes stability -- that is, an installed system keeps working in the same way for a long time. The tradeoff is slower releases and older software at certain points in a year compared to, say, Arch. On the other hand, compared to Arch, a Debian system has fewer times where a breaking change is likely to be introduced. So if you see what Debian's goals are, maybe this sort of thing makes some sense respective to them. Arch's goals are different and so different choices will flow out of that. The world probably needs both, so I'm not trying to slam Arch/BSD here. Actually - I ran into the versioning issue while packaging nncp. The blake3 package requires cpuid, and we had a newer cpuid in Debian than what the current blake3 was ready for. cpuid changed its interface a bit, so blake3 wouldn't compile. I patched blake3 in the Debian build to get it going (it was a 2-line fix, not really worth introducing multiple cpuid versions for), and also submitted the issue upstream. blake3 fixed the issue upstream as well, so I dropped our local Debian patch and upgraded to that version. So, you could pull in newer cpuid and blake3 into your tarballs, if you like. > And that frightens me, because if someone want to report a bug, > then it > have to report about dozen of dependencies and their exact > versions, > instead of reporting only of Go version and NNCP's one. Oh > well... Ahh, well we have a tool called reportbug that captures all of that with a bug report (or at least enough for a Debian maintainer to look it up in the project databases), and sends it to the Debian bug-tracking system. Then it's supposed to be the job of the Debian maintainer (me) to sort through that sort of thing before submitting it upstream (you). In this case, if there was any question, I would probably ask the submitter to try verifying the problem with a tarball-built NNCP before passing a report on to you. We've got the full build log of every package in the distro, for every architecture it may build on. For instance, here's the build log for blake3: https://buildd.debian.org/status/fetch.php?pkg=golang-lukechampine-blake3&arch=all&ver=1.1.6-1&stamp=1632155690&raw=0 And you can see the precise versions of go, cpuid, etc. that it built against. Though again, normally this is the job of the Debian maintainer to look at and sort through, not the upstream author. > personally I slowly moving more and more to GNU Stow type of > packages > management, manually building software :-) Haven't heard of Stow, but I have heard of Gentoo. I tried it for a spell. >>The uilive thing is a bit of an odd one; it was still in-tree in >>the git >>repo and it survived review so I guess it works for now :-) > > As I remember I copied it and modified. It is not original > version. Do > not remember why, but I believe to strip off various complicated > unneeded stuff. Makes sense. - John ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NNCP in Debian/Ubuntu Update 2021-09-20 21:13 ` John Goerzen @ 2021-09-21 10:10 ` Sergey Matveev 0 siblings, 0 replies; 6+ messages in thread From: Sergey Matveev @ 2021-09-21 10:10 UTC (permalink / raw) To: nncp-devel [-- Attachment #1: Type: text/plain, Size: 6968 bytes --] *** John Goerzen [2021-09-20 16:13]: >Fundamentally I can't just change that on my own, so our conversation here is >discussing something that "is what it is", regardless of whether we agree with >it. But, hey, I like conversation :-) I understand that of course :-). Here I just chatting around too. Just my two cents. >In Rust, there is a Cargo.toml, which defines the general requirements for >dependencies (ie, package X, version 0.2.*) and then a cargo.lock which >records the precise version and hash of the dependency used for a canonical >build. Go's go.mod is some kind of cargo.lock analogue -- where exact tag/commit of the dependant source code is set. And go.sum for assurance of exact source code contents. >The pro is, of course, that you can easily get a theoretically identical >build from anywhere. And that is completely worth of it! >And the con is that you may have dozens of different versions of the same >package in use on a system, making it difficult to patch bugs and security >issues. And one point of dynamically linked libraries is similar: easy upgrade path of some single globally shared library. But Go authors, Plan9 authors (same subset :-)) are heavily against dynamic linking concept (me too). Of course I can not put words in their mouths, but as I understand, there is assumption (I heavily support too) that software must be easily and quickly build and rebuilt. Of course each software package has to have the source code. So in some hypothetical Plan9, if there is need to upgrade its "OpenSSL" (actually there is no such task at all, because cryptography related tasks are just offloaded to separate (microservice) daemon -- single piece you need to update/rebuild), then you just rebuild every dependency. With Go, pure C (in most cases) -- compiling is very very fast. If someone needs to update Go itself (with its pretty feature-wide native libraries): just find all packages containing go.mod and run "go build" to rebuild literally every Go software in several minutes. If you need to update some dependency like golang.org/x/crypto, then probably "go get -u golang.org/x/crypto" in every Go's module will be enough, with later rebuilding. Problem of library updating and fixing is very minor one, not worth of using complexities like dynamic linking. Moreover dynamic linking does not give any guarantees that everything will go fine, because some mistakenly made backward incompatible API changes could break many things. So it is just some kind of good luck and still existing programming culture, that we can transparently do those upgrade paths. I was (and unfortunately still) Python programmer for many years and see that people are just *fully* incapable of providing correct dependency information at all. Python heavily relies (well, they try to move to Go/Rust-like lockfiles) on semantic versioning -- that is good. But it is enough to have only single package, that mistakenly (human factor) break it and everything could go wrong and won't install at all. So Go/Rust's behaviour is some kind of Holy Grail :-) -- this is the only thing that can guarantee working packages and dependencies. So in my opinion it is just much easier to rebuild all dependant software, than to blindly believe that yet another dynamic library update will go smoothly. No, people are incapable of doing this. >Tradeoffs abound. What's right for Debian isn't right for everybody. >Debian prizes stability -- that is, an installed system keeps working in the >same way for a long time. I used Debian for 6 or 7 years and truly remember its stability, comparing to all other popular distributions. I am definitely not a bleeding edge fan too :-). I installed Debian only once: 4.0 and made dist-upgrade/upgrade till 6.x versions. But... the world before modern dependency systems (like in Go/Rust/someone other I presume) was really so awful, that Debian's approach was good. But with Go... in my opinion it is hardly acceptable. And your example with blake3+cpuid shows its deficiency: Go's modules depend on exact version of software, so in theory nearly every library's commit have to be packaged as golang-github-XXX-gHASH, but you got some incompatibility and game is over :-). Either you keep each library's version somewhere under little chroot: /usr/local/lib/golang-github-XXX-gHASH and give "-I"-like points to Go compiler where to find exact versions, or you have to rewrite the software. I can not say "fix" software, because probably API change is not related to bug of anything similar. Basically Nix approach solves that problem: it keeps any kind of version under different paths, symlinking/including only necessary and required ones for each package independently. Moreover when you "give" different cpuid/blake3/whatever library version, it means that the program you build does not conform to its go.mod/go.sum. I would call it: broken build. Once I met funny thing: I used library that made incorrect/invalid tar archives, but I did not note that they were invalid and my code was fully dependant on that buggy behaviour. After library update I found that tar archives were not working anymore as expected. But I still anyway could use and build that bug-compatible code without any problems. Debian won't allow that, because program's author has no control over exact library version (or each version has to be separate package in separate directories, as in Nix). In my opinion Debian just breaks very good Go's approach of "do exactly as I say" (use exactly that code, with that checksum). Debian's built package may vary much from the original/vanilla version. Well, it is just another point of view how things should be done right, of course. >not really worth introducing multiple cpuid versions for Go assumes that any API change leads to creation of completely different library (namespace): github.com/klauspost/cpuid and github.com/klauspost/cpuid/v2 are completely different independent libraries. >you could pull in newer cpuid and blake3 into your tarballs, if you like. Yep, done: http://www.git.cypherpunks.ru/?p=nncp.git;a=commitdiff;h=dc6e5f1898c14e61985fd452c3fcf92914ca1207 >Ahh, well we have a tool called reportbug that captures all of that with a >bug report (or at least enough for a Debian maintainer to look it up in the >project databases), and sends it to the Debian bug-tracking system. Ah, good! >Haven't heard of Stow, but I have heard of Gentoo. I tried it for a spell. https://www.gnu.org/software/stow/ I used it initially for tracking my dotfiles (http://www.git.stargrave.org/?p=dotfiles.git;a=tree), but later more and more used for working with manually build software. It is just some kind of symlink manager :-) -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-09-21 10:11 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-20 13:05 NNCP in Debian/Ubuntu Update John Goerzen 2021-09-20 19:02 ` Sergey Matveev 2021-09-20 20:10 ` John Goerzen 2021-09-20 20:30 ` Sergey Matveev 2021-09-20 21:13 ` John Goerzen 2021-09-21 10:10 ` Sergey Matveev