<flokli>
whitequark[cis]: while trying to make the firmware reproduce, I tried with the nix-provided sdcc 4.2.0, but ran into a "?ASlink-Warning-Undefined Global 'sdcc_atomic_maybe_rollback' referenced by module 'usb'". Does that ring a bell?
<whitequark[cis]>
flokli: I don't understand what's going on with the firmware builds: the md5sum you posted doesn't match what's checked into the tree
<whitequark[cis]>
I can confirm that the glasgow.ihex you posted is different from the one in-tree, but when I run the deploy-firmware.sh script, I get a file identical to what's checked in
<whitequark[cis]>
which would have to be the case since it's impossible to land a commit changing firmware.ihex unless it reproduces
<flokli>
whitequark[cis]: I still had the glasgow tree I used to cross-check on x86_64-linux on that machine and was able to connect the dots, see the comment.
<whitequark[cis]>
flokli: okay, yeah, I figured it out now as well
<whitequark[cis]>
flokli: let me try and look into why it doesn't work with nix-provided sdcc
<whitequark[cis]>
sdcc_atomic_maybe_rollback is an implementation of C11 atomic operations
<whitequark[cis]>
we aren't using any C11 atomic operations
<whitequark[cis]>
so: assuming you've provided all the correct options, either debian is patching sdcc and fixing something broken, or nixos is patching sdcc and breaking something. or a secret third thing
<whitequark[cis]>
it looks like debian's only patch is fixing spelling errors in source code and changelogs. they must have lots of extra time
<whitequark[cis]>
* source code comments and changelogs.
<flokli>
yeah, they also have a almost 200 lines custom makefile in their packaging scripts.
<whitequark[cis]>
yes, it's upsetting
<whitequark[cis]>
wait, makefile?
<whitequark[cis]>
oh right debian/rules
<whitequark[cis]>
* right debian/rules is a makefile
<whitequark[cis]>
i actually did read through that and didn't find anything relevant
<whitequark[cis]>
re: your idea of using firmware checksums: probably the only way i'd ever do that is if i maintained an allowlist of firmware revisions that are accepted without an extra command line switch, but then some distro packager will patch out the check. i hate being in an adversarial relationship like that and my position is that if you're packaging my software at all you should respect the decisions that went into it.
<flokli>
whitequark[cis]: Let's keep going with the git commit id of the firmware/ dir baked into the firmware, I closed the PR changing that.
<whitequark[cis]>
flokli: alright. fwiw I think nix rebuilding the firmware from scratch to verify that it is identical is valuable and very much desired work
<whitequark[cis]>
it's just "blindly replace the firmware with something different if it turns out to be different" that I think is a net negative
<flokli>
I'm a bit exhausted from the discussion for today, I might pick it up in the future.
<whitequark[cis]>
so I'm happy to help you figure out why that doesn't work if it breaks
<flokli>
We can draft the nixpkgs bump PR and keep the old version from January in nixpkgs, which due to a packaging bug uses your firmware.
<whitequark[cis]>
thank you
<flokli>
I did pipe a lot of firmwares through diffoscope, and they're all just different. It's hard to figure out why.
<whitequark[cis]>
the easiest solution is to comment out that cp for the PR
<flokli>
probably, yes.
<whitequark[cis]>
I know some distributions have policies against that kind of thing but as far as I'm aware NixOS doesn't
<flokli>
We can set `sourceProvenance = with lib.sourceTypes; [ binaryBytecode ];`
<flokli>
and call it a day
<flokli>
But it means we cannot apply patches to backport fixes without including ihex changes in the patch.
<whitequark[cis]>
does anyone want to do that?
<whitequark[cis]>
most people get glasgow software by cloning a repository and tracking HEAD. there aren't releases and most likely there will never be releases. a lot of effort goes into maintaining backwards compatibility
<flokli>
s/binaryBytecode/binaryNativeCode/
figushki has joined #glasgow
<flokli>
If there's a fix in the firmware we'd want to apply, but we don't build the firmware from the updated sources, to get the firmware fixed we'd also need to include the updated binary code in the backport
<flokli>
well, if there's no tags/releases then this is probably less of an issue.
<whitequark[cis]>
glasgow is the software equivalent of a rolling distro
<flokli>
mmh
<flokli>
I guess I should open an issue about the libusb thing then
<whitequark[cis]>
given its scope and nature it's completely infeasible to do release branches, and simply doing releases doesn't seem meaningful: no commit is considered "more stable" than any other commit
<flokli>
yes
<whitequark[cis]>
re: libusb thing: yes, absolutely, I'm very interested in fixing that
<whitequark[cis]>
how do i get nix to give me sdcc 4.2.0?
<whitequark[cis]>
nix-shell -p sdcc gives me sdc 4.5.0
<flokli>
easiest would be to pick an old version of nixpkgs with 4.2.0
<flokli>
I did modify the glasgow package expression to modify the current packaging recipe to build 4.2.0 instead: https://paste.linuxlounge.net/AA
<whitequark[cis]>
ohhh right you can do that in nix
<flokli>
if it would have given me a firmware closer to what's in the repo, I would have happily added that. But with that the firmware build fails entirely
<whitequark[cis]>
we can probably just upgrade to 4.5.0
<flokli>
how? by switching from bookworm to something else?
<whitequark[cis]>
trixie, yeah. you are also entirely correct that i should've added a digest to the docker invocation
<whitequark[cis]>
that just leaves repository updates as the source of potential nondeterminism. back when I decided to use this script, I confirmed that Debian never updated sdcc in a stable release after it's been cut (presumably because there's no reason for anyone to do so)
<whitequark[cis]>
really I would have preferred to use Nix to get reproducible firmware builds because it's obviously much better suited to this task
<flokli>
what's preventing us?
<flokli>
I mean, going with the nix-shell shebang script and a pinned nixpkgs
<whitequark[cis]>
I want people running Debian or Ubuntu or something on a desktop to be able to do this
<whitequark[cis]>
this includes myself and also our CI infra
<flokli>
The shebang only requires Nix to be installed, not running on NixOS
<whitequark[cis]>
I actually have a nix-shell locally, I have no idea where I got it and it seems to be outdated
<flokli>
and there's even nix containers available, so we could wrap it into a nixos/nix container if we don't detect nix-build in $PATH.
<whitequark[cis]>
oh.
<flokli>
an outdated nix should still be fine as long as it's not too old to evaluate the nixpkgs we pin.
<whitequark[cis]>
that's an incredibly obvious thing in retrospect
<whitequark[cis]>
docker and nix are not mutually exclusive
<whitequark[cis]>
thanks, that's an excellent idea
<flokli>
The build sandbox is a bit less powerful if you're running inside a container, so it should still be a fallback only if you don't have nix-build natively.
<whitequark[cis]>
yeah, that's fine
<flokli>
But yeah, if you're willing to go that route, happy to take a look.
<whitequark[cis]>
yep, I'm not sure if you've found it but back before I merged the Docker solution, I have examined two: Docker and Nix
<whitequark[cis]>
(actually it looks like it was after I merged the Docker option and not before, I misremembered)
<flokli>
yes, this pins nixpkgs to a fixed rev, so should give you strong pining of sdcc and all of its deps.
<flokli>
I'd probably wrap this script with a second one that figures out whether nix-shell is in $PATH, and either executes it directly or inside a nixos/nix container.
<whitequark[cis]>
yep that's completely fine
<flokli>
and I'd probably bump the rev to f4a973b4a237153db1ed07f35cfa72b5b145276b, the latest commit on the current stable release.
<whitequark[cis]>
actually, have you tried running that script verbatim?
<whitequark[cis]>
I get bit-for-bit identical copy of the current checked-in firmware
<whitequark[cis]>
(you could probably just embed that into the derivation if you wanted to avoid tainting it)
<flokli>
yup, same hash for me. it puts the firmware.ihex into the wrong path, but otherwise same firmware
<whitequark[cis]>
right, so only the derivations using current nixpkgs (with sdcc 4.2 and 4.5) are broken in various ways
<whitequark[cis]>
do you get a different firmware file each time?
<whitequark[cis]>
I'm still really confused as to what would cause it; sdcc isn't known for nondeterminism in first place
<flokli>
I'd assume sdcc 4.2 -> 4.5 had some changes in codegen
<whitequark[cis]>
oh yes I wouldn't expect it to reproduce across sdcc versions
<whitequark[cis]>
I'm just confirming that sdcc 4.5 produces the same firmware each time for you
<flokli>
I kicked off a second build of the glasgow package, but I'd assume it to be identical
<whitequark[cis]>
can you tell me the sha1 ?
<whitequark[cis]>
oh that's a lot of nuisance warnings on 4.5
<flokli>
glasgow rev 27a68fc6a2fec37a22621d340c344990d06a4a3b built with nixpkgs produced firmware.ihex with sha1sum 3410e0144176bd0235f6ed58d361a4b728b025a0 on aarch64-linux, twice.
<flokli>
(running a build on x86_64-linux now)
GNUmoon has quit [Remote host closed the connection]
GNUmoon has joined #glasgow
<whitequark[cis]>
can you upload a firmware/build/glasgow.map file from the build? it's text
<whitequark[cis]>
and also firmware/build/glasgow.lk
<whitequark[cis]>
sdcc doesn't do anything clever during the compilation so i'd start looking at the linker issues first
<whitequark[cis]>
this looks to be the only meaningful difference
<whitequark[cis]>
my build is 14 bytes bigger for some reason
<flokli>
mm
<whitequark[cis]>
actually i think i'm not reading that diff right
<flokli>
same build (3410e0144176bd0235f6ed58d361a4b728b025a0) on x86_64-linux
<flokli>
your build is with sdcc 4.5 from debian?
<whitequark[cis]>
yes
<whitequark[cis]>
i think what's happening is that the constant (.rodata) area is 14 bytes bigger
<whitequark[cis]>
but it kind of doesn't make sense
<flokli>
wait, hold on
<whitequark[cis]>
none of the addresses inside are different, meaning it would be the last symbol that's the problem
<flokli>
this is the full nix build, which cannot shell out to git to determine the revision to bake into the firmware. What does the script use in that case?
<whitequark[cis]>
oh.
<flokli>
I can use the impure script from #469, with a bumped nixpkgs rev, and re-run the build
<whitequark[cis]>
it puts nothing in place of the revision
<whitequark[cis]>
and the reason it's 14 and not 8 is that my local work tree is dirty (I added --std=c23 to silence nuisance warnings)
<flokli>
so you'd come up with the same binary modulo the rev?
<whitequark[cis]>
the ihex file actually differs a lot, but I was looking at the map file
<whitequark[cis]>
let me do a proper binary diff
<flokli>
ok
<whitequark[cis]>
it is not exactly identical but it is very close
<whitequark[cis]>
ok, so actually we can just use trixie if i use a base image like debian:trixie-20250721-slim@sha256:cc92da07b99dd5c078cb5583fdb4ba639c7c9c14eb78508a2be285ca67cc738a
<whitequark[cis]>
i looked inside and it uses snapshot.debian.org to provide an immutable package index
<flokli>
did you check upstream never does GC these?
<whitequark[cis]>
yes
<flokli>
ok
<whitequark[cis]>
> The snapshot archive is a wayback machine that allows access to old packages based on dates and version numbers. It consists of all past and current packages the Debian archive provides.
<whitequark[cis]>
ok, so it's something in the environment
<whitequark[cis]>
this is the most baffling shit. I would have never expected ihex formatting to do this
<flokli>
TERM=, or my terminal size?
<whitequark[cis]>
$ echo $TERM $COLUMNS
<whitequark[cis]>
xterm-256color 108
<flokli>
hold on, that's inside the nix build, I cannot flip things there easily
<whitequark[cis]>
i tried changing COLUMNS and that didn't seem like it does anything
<whitequark[cis]>
looks like yours is 43 wide max, and mine is 75 wide max
<flokli>
I don't have enough context, but we cannot store and load raw binaries?
<whitequark[cis]>
not easily
<whitequark[cis]>
there are holes in the firmware image
<whitequark[cis]>
it would be possible to rearrange the build process so that the firmware that's actually stored is a flat binary, but it's more work than figuring out why no earth it does this
<whitequark[cis]>
in fact i'd rather add a python post-processing step to canonicalize the .ihex since that is like 1 line of code
<whitequark[cis]>
the CI only does the comparison when you change software/glasgow/hardware/firmware.ihex
<whitequark[cis]>
otherwise it would be impossible to check in any changes at all
<whitequark[cis]>
(given the self-referential nature of firmware builds)
<flokli>
Ah, that's how this is supposed to be read
<flokli>
Ok
<whitequark[cis]>
it also makes it possible to have in-tree changes to firmware that aren't automatically uploaded to everyone's device, which can be useful if you want to land a firmware change but still take a bit more time to test it
<whitequark[cis]>
it's a bit of a double edged sword
<flokli>
I'll update the packaging recipe when I'm back on the computer and check if it's the same (it should)
<whitequark[cis]>
thanks!
<flokli>
But I guess this means it's not a fully rolling distro, only the ones where the firmware is in sync may be packaged.
<_whitenotifier-8>
[glasgow] github-merge-queue[bot] created branch gh-readonly-queue/main/pr-981-4fe35360e71225a0ae7770a081d4c8e4259cc8e3 - https://github.com/GlasgowEmbedded/glasgow
<whitequark[cis]>
this is correct and for this reason I believe this possibility should not be used
<whitequark[cis]>
(i.e. every PR changing firmware should be immediately followed by a PR deploying it)
<flokli>
Could merge commits that are not fast-forwarded help make this workflow less painful?
<whitequark[cis]>
i don't use merge commits
figushki has joined #glasgow
<whitequark[cis]>
they make history absolutely hellish to browse
<whitequark[cis]>
it's fairly often that i give an answer of "look in the git history and do the same" to various maintenance questions, and i would wish trying to decipher which of the 20 parallel timelines in gitk they should follow on nobody
<_whitenotifier-8>
[glasgow] github-merge-queue[bot] created branch gh-readonly-queue/main/pr-981-4fe35360e71225a0ae7770a081d4c8e4259cc8e3 - https://github.com/GlasgowEmbedded/glasgow
<whitequark[cis]>
okay, that provides enough indication about the OS that i think additional info in version isn't needed
<whitequark[cis]>
thanks! seems good to go from my pov
<tpw_rules>
thought with both of you here, i think we can legally just always do version 0.1.0+g<hash> then we don't need extra nix maintenance to get the commits since root number
<flokli>
You can assume everyone with NixOS in that string to not use the pdm build, as that one fails due to .so file pointing to non-existing paths.
<tpw_rules>
sorry, version 0.1.dev0+g<hash>
<whitequark[cis]>
to further debug the timeouts i would like to see a wireshark usb communication dump
<flokli>
whitequark[cis]: I'll open the issue with what I have
<whitequark[cis]>
tpw_rules: yeah i'm fine with nixos setting .devX to .dev0
<tpw_rules>
thank you for your time as always
redstarcomrade has joined #glasgow
<whitequark[cis]>
^^
<tpw_rules>
flokli: i have glasgow hardware if you need an additional review
<whitequark[cis]>
flokli: i got really sidetracked by the mentions of firmware versions and API levels and such, and also that you've mentioned that unflashed devices work fine
<flokli>
whitequark[cis]: I didn't want to cause confusion, I was juggling a lot of things
<flokli>
Where did I mention unflashed devices work fine?
<flokli>
testing the PR
<whitequark[cis]>
> I was only able to reproduce when having the older firmware present; running glasgow flash --remove-firmware with the new glasgow version was not sufficient.
<flokli>
hmmh, then I must have thrown something together. I'm sorry if this threw you off the rails with debugging