<SiFuh>
sighook_: So you have been around for 200 years?
farkuhar has joined #opencrux
<farkuhar>
It looks like the CRUX 3.8 handbook does mention the 2GB system memory requirement (section 3.1, Supported Hardware). But if the minimum RAM is changing more rapidly than the distro release cadence, maybe it should be given a more prominent place in the release notes.
<farkuhar>
Release notes going back to CRUX 3.4 (that's as far as I've checked) never mention the minimum RAM requirement. They emphasize instead the changes in the core ports, such as the introduction of meson, cmake, or nftables.
<SiFuh>
farkuhar: Probably jaeger added it
<SiFuh>
Too many users were jumping on to #crux and complaining that something is broken in the install. Rather than addressing the real issue many of them tried offering guidance to the point I would step in and ask how much RAM do you have. Then I told jaeger that CRUX 3.7 cannot install on a machine with less than 2GB of RAM and the update ISO with extra ports can't run on a system with less than 4GB of RAM.
<SiFuh>
OpenCRUX will not have this issue because I have zero plans to ever load every port into system memory
<SiFuh>
That's why I have always liked the idea of booting the kernel with the rootfs and your USB stick has a second partition with all your ports on it. And you mount that.
<farkuhar>
Is that what the updated ISO was doing? Loading every port into system memory?
<SiFuh>
farkuhar: Yeah, from what I saw the ports are included in the rootfs. Making the loading rootfs take forever to copy over to RAM before you get a prompt.
<farkuhar>
Copying over the entire rootfs into RAM, that's dumb. I agree that a second partition, mounted after a minimal environment is up and running, is the way to go.
<SiFuh>
Well it wasn't dumb when Per set it up back in the day since we only had core.
<SiFuh>
In these modern times, it is indeed considered dumb
<farkuhar>
The practice of creating an updated ISO in between major releases ... is that a continuation of something that started with Per? Or did they start doing that after Per had already left the project?
<SiFuh>
Per released for major updates because there was no update feature at all.
<SiFuh>
You couldn't upgrade a core package that required config files back in the day because it might not boot well. So it was much simpler to just backup, fresh install and tweak the configs from the old backup to the new.
<SiFuh>
Keep in mind that prtutils didn't exist back then either.
<SiFuh>
You certainly do wake up early don't you farkuhar?
<farkuhar>
pkgutils, you mean? "no update feature" doesn't make sense on a source-based distro, only on a distro with binary packages. Even if early versions of pkgutils shipped with a pkgadd that didn't recognize the -u flag, you still had all the compilation tools to run `make install` by hand.
<SiFuh>
No I mean prt-get. And yes we had pkgmk pkgadd and pkgrm but it had nothing to handle config files
<farkuhar>
Hmm, so the pkgadd feature that exempts /etc files from upgrades was added much later?
<SiFuh>
Yes
<SiFuh>
Later down the line a very crude prt-get came out. I think that wasn't a Per thing. I think someone else wrote it (Python?) and eventually the concept became what is now prt-get we use today
<farkuhar>
Heh, no wonder it's still doing the poorly-optimized thing of recompiling the regex for every file (rather than precompiling the regex as braewoods patched it to do).
<SiFuh>
Well don't forget it was 25 years ago it was created ;-)
<SiFuh>
I was teaching Linux/Unix and Network security in a university in Thailand in those days :-P
<SiFuh>
Fsck I am old.
<farkuhar>
Well, it was only 5 years ago that filezilla lost a maintainer, got marked unmaintained, and eventually got dropped from contrib.
<SiFuh>
chinarulezzz!
<SiFuh>
I took a short hiatus from IRC and when I returned CRUX had changed a lot.
<farkuhar>
filezilla marked unmaintained 16:04:05 +0200 Fri 15 Jan 2021, then dropped one month later.
<SiFuh>
Which reminds me. What CRUX scripts use bash?
<farkuhar>
Are you asking about *sh as opposed to Perl, Python, Ruby, ..., or bash as opposed to POSIX sh? The latter question is more relevant if you want to target a lighter interpreter like dash.
<SiFuh>
Nah, I was thinking about things like the init scripts
<farkuhar>
Heh, even /etc/rc.conf relies on bash-specific features. If you want to target dash as the interpreter for /etc/rc, then you have to eradicate the use of arrays. We did a proof-of-concept sometime last year.
<SiFuh>
Well I don't plan on using anything but sh or OpenBSD's ksh.
<farkuhar>
The prt-utils README.md has a classification of the tools in that repo. https://git.crux.nu/tools/prt-utils/#sorted-by-language Maybe it would be worthwhile to extend this classification to all the distro-specific scripts.
<farkuhar>
I should probably update the "History of rewrites" section to mention the C90 revdep written by braewoods in 2015. The C++ rewrite came later.
<farkuhar>
s/C90/C99/
<farkuhar>
bash revdep was replaced by braewoods' C99 rewrite on Thu 12 March 2015, but then the C++ rewrite arrived less than a year later (Wed 24 February 2016).
<SiFuh>
Well, I want the basic system to be entirely POSIX. The extras can have bash and other stuff.
<farkuhar>
The 2016 commit advertises the C++ rewrite as offering "much improved speed". I doubt that all of the speed improvements can be explained by the choice of language. Compilers themselves might have improved in the intervening 11 months, but more importantly, a rewrite gave them a chance to think more carefully about the algorithm they were using.
<SiFuh>
I still haven't decided yet, but I was planning to have a separate repo for build tools.
<SiFuh>
So you meson, ninja, cmake, gcc, pkgutils and prt-get and stuff will be there.
<SiFuh>
That way someone can install a system and they can just look at the repo and see what they can uninstall quickly and efficiently
<farkuhar>
Actually I seem to recall that awk is part of the POSIX standard too. So the restriction you want to impose on the basic system needs to be clarified. Are you talking about scripts that currently have #!/bin/sh or #!/bin/bash as the shebang?
<SiFuh>
So if you build a server. You won't need those tools after it is running. And frankly it is not safe to leave that stuff installed on a server. However, pkgadd should probably exist but does it need to be installed?
<SiFuh>
#!/bin/sh is fine. Yes awk is POSIX, sed is not.
<sighook_>
farkuhar: revdep can be parallelized (it's on my todo), even if there's no insane bump due IO bottleneck. The better is to split it into the lib + microutility.
<sighook_>
as well as pkgutils (already split, just ugly now)
<farkuhar>
Well we should still be allowed to use sed in Pkgfiles, if the core-only system does not include compilation tools. The basic system can remain 100% POSIX in that case.
<SiFuh>
Anychance pkgutils cn be static?
zorz has joined #opencrux
<SiFuh>
Olutmies II
<sighook_>
SiFuh: who're you addressing the question to?
<farkuhar>
SiFuh: in the pkgutils Makefile that zorz wrote, you'll notice the clause #ifdef STATIC, which enables static linking to libarchive. But didn't you want to get rid of the libarchive dependency?
<SiFuh>
farkuhar: Yes, but I managed to to make it small it enough using libarchive so I scrapped the idea
<sighook_>
if pkgutils requires a universal packages handler (formats, xattr/acl metadata, streaming, portability), the libarchive has no real concurents here.
<farkuhar>
sighook_: no real alternative to libarchive? That reminds me of the time when an update broke bsdtar, and I had to edit pkgmk so that it would use p7zip instead (at least until I rebuilt libarchive and fixed the breakage).
<SiFuh>
zorz ran it through his AI to check the code.
<farkuhar>
pkgsize tells me that GNU tar takes up 508 kB of disk space here, while libarchive takes up 4248 kB. I can see why SiFuh wanted to change the dependency, even if only on the installation media.
<sighook_>
farkuhar: that's why may be is a good idea to compile bsdtar statically (if that fixes the poor maintaining practice), but not proves there's an alternative. in my humble opinion.
<sighook_>
yeah, taking into account your needs, there's definitely a reason to get rid of libarchive. i think. it's huge for core.
<farkuhar>
sighook_: The bsdtar breakage was many years ago. If I still have a copy of that modified pkgmk, the one that uses p7zip instead, I might be able to determine the date of the incident more precisely.
<SiFuh>
I've nothing against modifying or even a complete rewrite to make it C99 and efficient if anyone wants to give it a whirl.
<sighook_>
farkuhar: naaaah, don't bother yourself, i totally understood your argument, and it has a point.
<SiFuh>
In fact, what if prt-get and pkgutils were merged into one program?
<sighook_>
SiFuh: bad idea, imho.
<sighook_>
but it's up to you.
<SiFuh>
Reason?
<sighook_>
my view on future toolchain: pkgutils -> split (utils + lib), revdep -> split (utils + lib), pkgman (prt-get) (optional compilation with libpkgutils + librevdep.
<SiFuh>
I see
<sighook_>
so, we enchance the toolchain, leaving the hackability of rewriting the pkgadd/pkgrm hooks.
<SiFuh>
Hmm, agreed
<sighook_>
think of prt-get as orchestrator around tiny utilities. optional, for speed, it can be compiled for speed with default libpkgutils/lbirevdep, but while defining in /etc/prt-get.conf addcommand, rmcommand, it switch to custom wrappers.
<sighook_>
i think it's good architecturally.
<sighook_>
also, it gives us a lot of tiny small utilities that is easy to maintain and comprehend.
<sighook_>
they can be compound in unix way, can be used separately/directly.
<farkuhar>
sighook_ also said it was a bad idea architecturally to give prt-get the feature of synchronizing the ports trees using the drivers under /etc/ports. Separate tools for separate tasks is more aligned with the Unix philosophy.
<SiFuh>
Hmm farkuhar You are WinTurd user. How because this fscking OS installed to disk?
<SiFuh>
I am running a clonezilla backup of the woman's laptop and it says 257GB and she says she hasn't much on it.
<farkuhar>
My view is that if prt-get is already an "orchestrator around tiny utilities", why not let it orchestrate the drivers under /etc/ports? The main downside of the way I implemented a `prt-get sync` command is that it currently reads the config file of an ostensibly different program (/usr/bin/ports).
<SiFuh>
What the hell did I just say?
<SiFuh>
Hmm farkuhar You are a WinTurd user. How much space does this fscking OS use when it is installed to a disk.
<farkuhar>
SiFuh just said something about a 257GB hard disk on some woman's laptop. I don't have any Windows installations currently within easy reach. I'd have to visit my workplace to look up the information he's requesting.
<SiFuh>
Hahaha
<farkuhar>
Or he could just ask ChatGPT for the answer.
<SiFuh>
How much disk space does WInTurd 11 use when installed? Gemini: Fresh Install Size: Roughly 20 GB to 30 GB.
<SiFuh>
A roughly 10GB difference. That is really rough.
<SiFuh>
Gemini says it could be the hiberfil.sys
<SiFuh>
And that I should have ran some weird-ass command powercfg /h off first
<SiFuh>
manage-bde -status And to check if bitlocker is off
<farkuhar>
After you started appending the Roman numerals, I'm more inclined to think of an Egyptian pharaoh when reading "Olutmies II". Kudos for discovering this extra layer of obfuscation.
<sighook_>
farkuhar: maybe it suits for you and OpenCRUX, sincerely, i dunno. i'm reflecting my view on zpln. i thought now to reflect the view, but it requires an email-like message, about architectural perspective and problems it gives. So, let's just summarize: from unix/kiss perspective, the issue is not "more code" but "more kinds of responsibility". adding libpkgutils library calls is a vertical extension
<sighook_>
within prt-get domain. adding repository syncing is a horizontal expansion into a different domain. just a drift toward monolith with fuzzy boundaries.
<SiFuh>
sighook_?
<SiFuh>
We got it!
<SiFuh>
Hahaha
<sighook_>
we have a thread with farkuhar about "prt-get sync" patch he remplemented in his prt-get branch.
<farkuhar>
prt-get already has responsibilities involving the ports collections, to the extent that it scans Pkgfiles to resolve dependencies (limiting its search to prtdir directives in prt-get.conf) and leaves build artifacts in the ports tree (if PKGMK_PACKAGE_DIR is not redefined). It's not much of a horizontal extension to give prt-get the ability to sync the active ports collections.
<farkuhar>
It could be argued that prt-get's current access to the ports collections is read-only, except when a spawned pkgmk process leaves build artifacts. But the sync operation also involves a spawned process (under /etc/ports/drivers), so the precedent of a spawned pkgmk process is not irrelevant.
<farkuhar>
Architecturally, it's not the cleanest design to keep using the filename suffix of /etc/ports/$repo.$driver as an independent toggle of what gets synced with upstream. But it's too inflexible to insist that all the prtdir directives in prt-get.conf correspond to repos that need syncing. Some active repos you just want to maintain locally.
<sighook_>
lets define the roles explicitly: pkgmk / pkgadd / pkgrm are executors. They operate on local: pkgmk - on package source (port), pkgadd - already materialized package, pkgrm - already installed package. They do not know where it came from. prt-get is an orchestrator. it adds policy: dependency resolution, ordering, conflict handling, configuration interpretation, etc. Its input assumption is "a set of
<sighook_>
ports collections exists locally". ports -u (stupid name, lol) is a provisioner. it manages state acquisition: remotes, transport, credentials, transport, mirroring strategy, and filesystem layout under /usr/ports. Those are three distinct layers. They key question is whether prt-get should expand "upwards" into provisioning, or whether provisioning should remain an external concern.
<sighook_>
sorry if i mix current zpln state and outdated crux info i have.
<sighook_>
haven't used crux a years, just what i remember.
<sighook_>
so i can write prt-get (while refer to pkgman), pkgmk which is not use pkgadd under the hood, but is separated explicitly, etc.
<SiFuh>
I used it two days ago
<sighook_>
but in general, my design expectation don't think goes too far from current crux state.
<sighook_>
so, examples are valid.
<sighook_>
for example, right now pkgman doesnt give a fluck how /usr/ports was populated: git, rsync, tarballs, zfs snapshots, nfs mounts, sshfs mounts, all satisfy the same interface: a directory tree.
<sighook_>
leaving updating policy (git pull --rebase for devs, git fetch + reset on user, branches, tags), periodic cron-based syncing, manual syncing review, builds again frozen trees, multiple collections sharing the same /usr/ports just into a different domain.
<farkuhar>
sighook_: the updating policy is delegated to the executable files under /etc/ports/drivers, both in the bash script /usr/bin/ports and in my `prt-get sync` command. You could say that my forked prt-get also doesn't give a fluck how the ports collections were populated, the appropriate driver can be selected independently on a per-repo basis.
<farkuhar>
Arguably even `ports -u` is not a provisioner, it's merely the orchestrator for the executables under /etc/ports/drivers, selected according to the suffix of /etc/ports/$repo.$driver. Those executables under /etc/ports/drivers are the real provisioners.
<farkuhar>
The fact that /usr/bin/ports can be invoked with other modes of operation (listing and diff, both already implemented in prt-get) is perhaps what triggered the desire to fold its remaining functionality into prt-get.
<sighook_>
yeah, i remember that patch, i wrote it and know how it operates. deepening into who actually are providers is speculative, i looked from roles perspective not implementation, because we can say /etc/ports/drivers are not providers: git, rsync, csv are.
<sighook_>
farkuhar: exactly, ports utility is vague. i completely get rid of it. so, my perspective is not "keeping synchronization while get rid of ports(8) redundancies".
<sighook_>
s/csv/cvs/
<farkuhar>
Maybe it was a poor design decision to have pkgmk respect the flags -i and -u (install and upgrade), rather than limiting its role to package building and tarball creation. But as long as those flags continue being supported by pkgmk, to me it made sense to let prt-get delegate the eventual pkgadd invocation to the spawned pkgmk process.
<farkuhar>
The only problem with such delegation is that pkgmk didn't originally have flags for customizing the addcommand (hard-coded as /usr/bin/pkgadd) or the target rootfs. Add those flags to pkgmk, and then prt-get does not need to concern itself with micromanaging both the pkgmk process and a subsequent pkgadd process.
<sighook_>
yeah. it makes sense. it breaks utility separation, wrappering, precise error handling, and staging during prt-get transaction, and so, and so... but in general. c'mon, it's crux. it fcked up as SiFuh said, long long ago.
<sighook_>
farkuhar: oooooh, i see zorz strategy here :-P
<sighook_>
instead of clearly separating pkgmk into his role.
<farkuhar>
pkgmk already has separate error codes to distinguish "error while running build()" from "error while installing the package". If precise error handling is desired, it can easily be achieved even with prt-get relinquishing control of the pkgadd step.
<farkuhar>
E_BUILD=8 and E_INSTALL=9, at the top of /usr/bin/pkgmk.
<sighook_>
farkuhar: yeah, but that's error handling is broken architecture. utility which makes software package has an error code "error while installing package", cmon :-D
<sighook_>
i mean, cmon my brother in Christ, it's just broken crux legacy.
<sighook_>
it's totally fine if you want to preserve that. but that's not design.
<farkuhar>
Now I'm curious whether the -i and -u flags were around from the very beginning, or whether they were added later.
<farkuhar>
The fragility of prt-get micromanaging the pkgadd invocation is that it has to know the precise path to the built package. If users are clever enough, they can write a Pkgfile with enough bashisms so that prt-get is unable to determine the correct filename, while pkgmk (as a bash script) has no problem locating the package it built.
<sighook_>
all that kind of flaws are fixed in zeppe-lin. i mean, these tools are published and available. if you'll find it useful you can use it, and we can unify our efforts on tools we share common view to improve them for both, you from OpenCRUX perspective, me from Zeppe-Lin.
<sighook_>
i'm not pushing you to zeppe-lin, i'm just saying, we can share our efforts of something we have common, while keeping separate all we don't have.
<sighook_>
because we are small community around Per legacy.
<farkuhar>
Delegating the pkgadd invocation to pkgmk is the more robust way to work around such user cleverness (too many bashisms that prt-get cannot resolve). I agree that it violates separation of concerns, but the fragility was a more important issue for me.
<farkuhar>
Of course, if you rewrite pkgmk in C99 and loudly announce the deprecation of all bashisms, then users have no excuse for tinkering with Pkgfiles to try triggering such breakage. But not everyone will read the documentation; most people will just inspect existing Pkgfiles, see what looks like bash code, and feel entitled to employ any bashisms they know.
<sighook_>
lol, yepp
<sighook_>
duck typing without duck
<sighook_>
but i will not dig into pkgmkc99 architecture, because if i have so many concerns about crux tooling architecture... i mean, i'll just leave the soul alone.
<farkuhar>
With Vitaly's approach, prt-get remains the overall orchestrator of spawned processes, and pkgmk only needs to recognize one new flag (not two, for addcommand and target-root). Error handling becomes easier when orchestration is kept at the top level of the process tree.
<farkuhar>
As for whether it was the wrong decision to have pkgmk recognize the -i and -u flags at all, therealfun appears to be in agreement that such design violates separation of concerns. His "micro pkgmk" removes support for all flags except -c: https://gitlab.com/therealfun/oprt/-/blob/master/src/upkgmk?ref_type=heads
<farkuhar>
"clean up your own mess" must be the principle that therealfun is using to justify continued support for the -c flag. On the basis of separation of concerns, one might argue that cleaning up the repo is a job we can reserve for other tools (e.g., prtsweep). And then his "micro pkgmk" could be made even smaller.
<farkuhar>
It's not practical to enforce a universal policy that for every feature added, two features have to be removed (with the goal of countering the tendency toward bloat). But in the case of Vitaly's patch, adding support for the -dc flag could be combined with dropping support for -i and -u, since the resulting pkgmk is intended to be invoked by prt-get.