public inbox for nncp-devel@lists.cypherpunks.ru
Atom feed
* NNCP in Debian/Ubuntu Update
@ 2021-09-20 13:05 John Goerzen
  2021-09-20 19:02 ` Sergey Matveev
  0 siblings, 1 reply; 6+ messages in thread
From: John Goerzen @ 2021-09-20 13:05 UTC (permalink / raw)
  To: nncp-devel

Hello everyone,

At long last, NNCP has been accepted into Debian unstable.  This 
is great news.  It means NNCP will be in future Debian (and 
derivatives such as Ubuntu) releases.

From here, I hope to get it migrated to testing in the next couple 
of weeks.  Once in testing, I can build it for buster and bullseye 
backports.  This will make it apt-get installable on lots of 
current machines, including Raspberry Pis and anyone running the 
current or prior release of Debian.

- John

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NNCP in Debian/Ubuntu Update
  2021-09-20 13:05 NNCP in Debian/Ubuntu Update John Goerzen
@ 2021-09-20 19:02 ` Sergey Matveev
  2021-09-20 20:10   ` John Goerzen
  0 siblings, 1 reply; 6+ messages in thread
From: Sergey Matveev @ 2021-09-20 19:02 UTC (permalink / raw)
  To: nncp-devel

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

Greetings!

*** John Goerzen [2021-09-20 08:05]:
>At long last, NNCP has been accepted into Debian unstable.

Wonderful! Thank you!

Just for the interest I looked at https://salsa.debian.org/go-team/packages/nncp.git
I have never dealt with Debian packages (well, 15+ years ago played only
with the one), so just my misunderstanding, but it seems that NNCP is
not built from the tarball? I see some dependencies on external
Go-related packages and documentation building in debian/rules file. I
remind that NNCP's tarballs contain all Go dependencies, that are
statically linked in anyway. And they contain both Info and HTML
documentation.

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NNCP in Debian/Ubuntu Update
  2021-09-20 19:02 ` Sergey Matveev
@ 2021-09-20 20:10   ` John Goerzen
  2021-09-20 20:30     ` Sergey Matveev
  0 siblings, 1 reply; 6+ messages in thread
From: John Goerzen @ 2021-09-20 20:10 UTC (permalink / raw)
  To: Sergey Matveev; +Cc: nncp-devel

On Mon, Sep 20 2021, Sergey Matveev wrote:

> Greetings!
>
> *** John Goerzen [2021-09-20 08:05]:
>>At long last, NNCP has been accepted into Debian unstable.
>
> Wonderful! Thank you!
>
> Just for the interest I looked at 
> https://salsa.debian.org/go-team/packages/nncp.git
> I have never dealt with Debian packages (well, 15+ years ago 
> played only
> with the one), so just my misunderstanding, but it seems that 
> NNCP is
> not built from the tarball? I see some dependencies on external
> Go-related packages and documentation building in debian/rules 
> file. I
> remind that NNCP's tarballs contain all Go dependencies, that 
> are
> statically linked in anyway. And they contain both Info and HTML
> documentation.

Yes.  Actually this is the reason it took me so long to get it in 
Debian; I had to learn about Go building and stuff.

So Debian has a very long-standing policy, going back decades, of 
not building dependencies from in-tree.  Even back in the C days, 
a lot of packages would require other libraries and include them 
in their tarballs.  That caused a number of problems at the 
distribution scale.  For instance, if there's a new OpenSSL vuln 
and various packages build from OpenSSL in-tarball, now instead of 
patching one thing, you first have to figure out where all it's 
embedded (a challenge itself) and then patch and rebuild all of 
those.  The same goes for bugs in OpenSSL, etc.  It's not 
particularly sustainable.

There is also another long-standing policy that every Debian 
package must be buildable using only what is in Debian, and 
without network access.  This is enforced at the CI level.  The 
rationale for this includes several aspects: 1) A build that 
downloads things from the Internet can break if those things go 
away; 2) it makes it easier to inject malware into the system; 3) 
it makes it harder to audit for license compliance; 4) it can lead 
to unintentional violations of the Debian Free Software Guidelines 
(if sources are downloaded but not distributed, this can also be a 
GPL violation).  So the Go and Rust ecosystems (plus others like 
Python) don't use the go/cargo/pip build mechanism to just 
download stuff off the Internet; they require that the 
dependencies already exist in Debian, and build using only those.

I did initially try to just use the tarball, but that was 
(somewhat predictably) rejected for these reasons.  So, I made 
separate packages for all the things that weren't already in 
Debian (blake3, balloon, etc) and got those into the archive. 
Then NNCP could go in. 
https://go-team.pages.debian.net/packaging.html talks about this.

The tarball is absolutely the right thing for users.  I like what 
you've done there.  It just wasn't the most suitable for a 
distribution (or, shall we say, a distribution with Debian's 
policies).

There is automated tooling to bring go stuff into Debian, since 
this is far from the only such case.  It is centered around the go 
convention of using git.  It mostly worked, here.  I had to patch 
up a few things but generally it worked well.  In fact, I do have 
a 7.7.0 release prepared -- the upgrade was one command -- which I 
will upload probably tomorrow once all of its dependencies have 
hit the mirrors.

The uilive thing is a bit of an odd one; it was still in-tree in 
the git repo and it survived review so I guess it works for now 
:-)

I also had to do a bit of hacking around the source tree; I don't 
quite understand it, but there was something relating to nncp/v7 
that the automated tooling didn't quite deal well with (you can 
see this in the debian/rules file).

So anyway, the long and short of it is, it is done, in the Debian 
way, and is working and will move ahead!

- John

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NNCP in Debian/Ubuntu Update
  2021-09-20 20:10   ` John Goerzen
@ 2021-09-20 20:30     ` Sergey Matveev
  2021-09-20 21:13       ` John Goerzen
  0 siblings, 1 reply; 6+ messages in thread
From: Sergey Matveev @ 2021-09-20 20:30 UTC (permalink / raw)
  To: nncp-devel

[-- Attachment #1: Type: text/plain, Size: 2187 bytes --]

*** John Goerzen [2021-09-20 15:10]:
>So Debian has a very long-standing policy, going back decades, of not
>building dependencies from in-tree.

Hmm... I probably can understand that for dynamically linked
executables, but that seems strange for statically linked ones,
that Go does by default. Moreover, Go's go.mod/go.sum are tied
not to some abstract version of the library, but literally to its
exact commit/tag. And even in one Go program you can easily use
different versions of the same library, that will force you to
install golang-whatever-* of various versions. All of that seems
very strange to me, but it is Debian's decision, ok :-)

>There is also another long-standing policy that every Debian package must be
>buildable using only what is in Debian, and without network access.

That is good, without any doubts!

>It just wasn't the most suitable for a distribution (or, shall
>we say, a distribution with Debian's policies).

And that frightens me, because if someone want to report a bug, then it
have to report about dozen of dependencies and their exact versions,
instead of reporting only of Go version and NNCP's one. Oh well...
personally I slowly moving more and more to GNU Stow type of packages
management, manually building software :-)

>The uilive thing is a bit of an odd one; it was still in-tree in the git
>repo and it survived review so I guess it works for now :-)

As I remember I copied it and modified. It is not original version. Do
not remember why, but I believe to strip off various complicated
unneeded stuff.

>I also had to do a bit of hacking around the source tree; I don't quite
>understand it, but there was something relating to nncp/v7 that the
>automated tooling didn't quite deal well with (you can see this in the
>debian/rules file).

Strange, have no ideas where "src/go.cypherpunks.ru/nncp/v7/src" path
comes from. But yeah, last "src/" is not needed.

>So anyway, the long and short of it is, it is done, in the Debian way, and
>is working and will move ahead!

Cool!

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NNCP in Debian/Ubuntu Update
  2021-09-20 20:30     ` Sergey Matveev
@ 2021-09-20 21:13       ` John Goerzen
  2021-09-21 10:10         ` Sergey Matveev
  0 siblings, 1 reply; 6+ messages in thread
From: John Goerzen @ 2021-09-20 21:13 UTC (permalink / raw)
  To: Sergey Matveev; +Cc: nncp-devel


On Mon, Sep 20 2021, Sergey Matveev wrote:

> *** John Goerzen [2021-09-20 15:10]:
>>So Debian has a very long-standing policy, going back decades, 
>>of not
>>building dependencies from in-tree.
>
> Hmm... I probably can understand that for dynamically linked
> executables, but that seems strange for statically linked ones,
> that Go does by default. Moreover, Go's go.mod/go.sum are tied
> not to some abstract version of the library, but literally to 
> its
> exact commit/tag. And even in one Go program you can easily use
> different versions of the same library, that will force you to
> install golang-whatever-* of various versions. All of that seems
> very strange to me, but it is Debian's decision, ok :-)

Fundamentally I can't just change that on my own, so our 
conversation here is discussing something that "is what it is", 
regardless of whether we agree with it.  But, hey, I like 
conversation :-)

I'm not as familiar with the Go build, but if I could express this 
in terms of rust:

In Rust, there is a Cargo.toml, which defines the general 
requirements for dependencies (ie, package X, version 0.2.*) and 
then a cargo.lock which records the precise version and hash of 
the dependency used for a canonical build.

There are pros and cons to this...

The pro is, of course, that you can easily get a theoretically 
identical build from anywhere.

And the con is that you may have dozens of different versions of 
the same package in use on a system, making it difficult to patch 
bugs and security issues.

So any person can come down on one side or the other of that, and 
I totally understand that.  Debian as a project has chosen where 
it comes down, and carries the pros and cons with it.  FreeBSD and 
Arch probably come down differently, and live with the pros and 
cons of their choices too.

One thing that flows out of these decisions is the possibility of 
unattended-upgrades existing in Debian, but really not so much in 
FreeBSD ports or Arch.  unattended-upgrades basically can patch 
your system for security bugs automatically from cron.  That's not 
for everyone, but I love it.  It's possible primarily because 
Debian backports security fixes to the version in stable.  That 
is, you get a narrowly-targeted patch that has a low risk of 
introducing issues because it isn't pulling in major new versions 
of software and such.  But the choice here also helps, because if 
the distro follows cargo.lock of the go equivalent, that can 
preclude the possibility of narrowly-targeted patches and also 
expand the work of the security team to unsustainable levels.

Tradeoffs abound.  What's right for Debian isn't right for 
everybody.  Debian prizes stability -- that is, an installed 
system keeps working in the same way for a long time.  The 
tradeoff is slower releases and older software at certain points 
in a year compared to, say, Arch.  On the other hand, compared to 
Arch, a Debian system has fewer times where a breaking change is 
likely to be introduced.  So if you see what Debian's goals are, 
maybe this sort of thing makes some sense respective to them. 
Arch's goals are different and so different choices will flow out 
of that.  The world probably needs both, so I'm not trying to slam 
Arch/BSD here.

Actually - I ran into the versioning issue while packaging nncp. 
The blake3 package requires cpuid, and we had a newer cpuid in 
Debian than what the current blake3 was ready for.  cpuid changed 
its interface a bit, so blake3 wouldn't compile.  I patched blake3 
in the Debian build to get it going (it was a 2-line fix, not 
really worth introducing multiple cpuid versions for), and also 
submitted the issue upstream.  blake3 fixed the issue upstream as 
well, so I dropped our local Debian patch and upgraded to that 
version.  So, you could pull in newer cpuid and blake3 into your 
tarballs, if you like.

> And that frightens me, because if someone want to report a bug, 
> then it
> have to report about dozen of dependencies and their exact 
> versions,
> instead of reporting only of Go version and NNCP's one. Oh 
> well...

Ahh, well we have a tool called reportbug that captures all of 
that with a bug report (or at least enough for a Debian maintainer 
to look it up in the project databases), and sends it to the 
Debian bug-tracking system.  Then it's supposed to be the job of 
the Debian maintainer (me) to sort through that sort of thing 
before submitting it upstream (you).  In this case, if there was 
any question, I would probably ask the submitter to try verifying 
the problem with a tarball-built NNCP before passing a report on 
to you.

We've got the full build log of every package in the distro, for 
every architecture it may build on.  For instance, here's the 
build log for blake3:

https://buildd.debian.org/status/fetch.php?pkg=golang-lukechampine-blake3&arch=all&ver=1.1.6-1&stamp=1632155690&raw=0

And you can see the precise versions of go, cpuid, etc. that it 
built against.  Though again, normally this is the job of the 
Debian maintainer to look at and sort through, not the upstream 
author.

> personally I slowly moving more and more to GNU Stow type of 
> packages
> management, manually building software :-)

Haven't heard of Stow, but I have heard of Gentoo.  I tried it for 
a spell.

>>The uilive thing is a bit of an odd one; it was still in-tree in 
>>the git
>>repo and it survived review so I guess it works for now :-)
>
> As I remember I copied it and modified. It is not original 
> version. Do
> not remember why, but I believe to strip off various complicated
> unneeded stuff.

Makes sense.

- John

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: NNCP in Debian/Ubuntu Update
  2021-09-20 21:13       ` John Goerzen
@ 2021-09-21 10:10         ` Sergey Matveev
  0 siblings, 0 replies; 6+ messages in thread
From: Sergey Matveev @ 2021-09-21 10:10 UTC (permalink / raw)
  To: nncp-devel

[-- Attachment #1: Type: text/plain, Size: 6968 bytes --]

*** John Goerzen [2021-09-20 16:13]:
>Fundamentally I can't just change that on my own, so our conversation here is
>discussing something that "is what it is", regardless of whether we agree with
>it.  But, hey, I like conversation :-)

I understand that of course :-). Here I just chatting around too. Just
my two cents.

>In Rust, there is a Cargo.toml, which defines the general requirements for
>dependencies (ie, package X, version 0.2.*) and then a cargo.lock which
>records the precise version and hash of the dependency used for a canonical
>build.

Go's go.mod is some kind of cargo.lock analogue -- where exact
tag/commit of the dependant source code is set. And go.sum for
assurance of exact source code contents.

>The pro is, of course, that you can easily get a theoretically identical
>build from anywhere.

And that is completely worth of it!

>And the con is that you may have dozens of different versions of the same
>package in use on a system, making it difficult to patch bugs and security
>issues.

And one point of dynamically linked libraries is similar: easy upgrade
path of some single globally shared library. But Go authors, Plan9
authors (same subset :-)) are heavily against dynamic linking concept
(me too). Of course I can not put words in their mouths, but as I
understand, there is assumption (I heavily support too) that software
must be easily and quickly build and rebuilt. Of course each software
package has to have the source code. So in some hypothetical Plan9, if
there is need to upgrade its "OpenSSL" (actually there is no such task
at all, because cryptography related tasks are just offloaded to
separate (microservice) daemon -- single piece you need to
update/rebuild), then you just rebuild every dependency. With Go, pure C
(in most cases) -- compiling is very very fast. If someone needs to
update Go itself (with its pretty feature-wide native libraries): just
find all packages containing go.mod and run "go build" to rebuild
literally every Go software in several minutes. If you need to update
some dependency like golang.org/x/crypto, then probably "go get -u
golang.org/x/crypto" in every Go's module will be enough, with later
rebuilding. Problem of library updating and fixing is very minor one,
not worth of using complexities like dynamic linking.

Moreover dynamic linking does not give any guarantees that everything
will go fine, because some mistakenly made backward incompatible API
changes could break many things. So it is just some kind of good luck
and still existing programming culture, that we can transparently do
those upgrade paths. I was (and unfortunately still) Python programmer
for many years and see that people are just *fully* incapable of
providing correct dependency information at all. Python heavily relies
(well, they try to move to Go/Rust-like lockfiles) on semantic
versioning -- that is good. But it is enough to have only single
package, that mistakenly (human factor) break it and everything could go
wrong and won't install at all. So Go/Rust's behaviour is some kind of
Holy Grail :-) -- this is the only thing that can guarantee working
packages and dependencies. So in my opinion it is just much easier to
rebuild all dependant software, than to blindly believe that yet another
dynamic library update will go smoothly. No, people are incapable of
doing this.

>Tradeoffs abound.  What's right for Debian isn't right for everybody.
>Debian prizes stability -- that is, an installed system keeps working in the
>same way for a long time.

I used Debian for 6 or 7 years and truly remember its stability,
comparing to all other popular distributions. I am definitely not a
bleeding edge fan too :-). I installed Debian only once: 4.0 and made
dist-upgrade/upgrade till 6.x versions.

But... the world before modern dependency systems (like in
Go/Rust/someone other I presume) was really so awful, that Debian's
approach was good. But with Go... in my opinion it is hardly acceptable.
And your example with blake3+cpuid shows its deficiency: Go's modules
depend on exact version of software, so in theory nearly every library's
commit have to be packaged as golang-github-XXX-gHASH, but you got some
incompatibility and game is over :-). Either you keep each library's
version somewhere under little chroot:
/usr/local/lib/golang-github-XXX-gHASH and give "-I"-like points to Go
compiler where to find exact versions, or you have to rewrite the
software. I can not say "fix" software, because probably API change is
not related to bug of anything similar. Basically Nix approach solves
that problem: it keeps any kind of version under different paths,
symlinking/including only necessary and required ones for each package
independently. Moreover when you "give" different cpuid/blake3/whatever
library version, it means that the program you build does not conform to
its go.mod/go.sum. I would call it: broken build. Once I met funny
thing: I used library that made incorrect/invalid tar archives, but I
did not note that they were invalid and my code was fully dependant on
that buggy behaviour. After library update I found that tar archives
were not working anymore as expected. But I still anyway could use and
build that bug-compatible code without any problems. Debian won't allow
that, because program's author has no control over exact library version
(or each version has to be separate package in separate directories, as
in Nix). In my opinion Debian just breaks very good Go's approach of "do
exactly as I say" (use exactly that code, with that checksum). Debian's
built package may vary much from the original/vanilla version. Well, it
is just another point of view how things should be done right, of course.

>not really worth introducing multiple cpuid versions for

Go assumes that any API change leads to creation of completely different
library (namespace): github.com/klauspost/cpuid and github.com/klauspost/cpuid/v2
are completely different independent libraries.

>you could pull in newer cpuid and blake3 into your tarballs, if you like.

Yep, done: http://www.git.cypherpunks.ru/?p=nncp.git;a=commitdiff;h=dc6e5f1898c14e61985fd452c3fcf92914ca1207

>Ahh, well we have a tool called reportbug that captures all of that with a
>bug report (or at least enough for a Debian maintainer to look it up in the
>project databases), and sends it to the Debian bug-tracking system.

Ah, good!

>Haven't heard of Stow, but I have heard of Gentoo.  I tried it for a spell.

https://www.gnu.org/software/stow/
I used it initially for tracking my dotfiles
(http://www.git.stargrave.org/?p=dotfiles.git;a=tree), but later more
and more used for working with manually build software. It is just some
kind of symlink manager :-)

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-09-21 10:11 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-20 13:05 NNCP in Debian/Ubuntu Update John Goerzen
2021-09-20 19:02 ` Sergey Matveev
2021-09-20 20:10   ` John Goerzen
2021-09-20 20:30     ` Sergey Matveev
2021-09-20 21:13       ` John Goerzen
2021-09-21 10:10         ` Sergey Matveev