equwal has quit [Remote host closed the connection]
equwal has joined #commonlisp
Ruby has joined #commonlisp
kchanqvq14 has joined #commonlisp
mange has joined #commonlisp
kchanqvq16 has quit [Ping timeout: 250 seconds]
equwal has quit [Remote host closed the connection]
fosskers has joined #commonlisp
fosskers has quit [Remote host closed the connection]
vidak has quit [Quit: Konversation terminated!]
kchanqvq14 is now known as kchanqvq
Pixel_Outlaw has joined #commonlisp
bpanthi977 has quit [Ping timeout: 260 seconds]
pestctrl has quit [Ping timeout: 260 seconds]
bpanthi977 has joined #commonlisp
admich1 has quit [Remote host closed the connection]
nullbath has left #commonlisp [Killed buffer]
admich1 has joined #commonlisp
admich1 has quit [Read error: Connection reset by peer]
bpanthi978 has joined #commonlisp
bpanthi977 has quit [Ping timeout: 260 seconds]
bpanthi978 is now known as bpanthi977
equwal has joined #commonlisp
equwal has quit [Remote host closed the connection]
equwal has joined #commonlisp
kchanqvq24 has joined #commonlisp
kchanqvq has quit [Ping timeout: 250 seconds]
<equwal>
i like lisp
kchanqvq24 is now known as kchanqvq
scymtym has quit [Ping timeout: 248 seconds]
istewart has joined #commonlisp
scymtym has joined #commonlisp
kchanqvq21 has joined #commonlisp
triffid has quit [Remote host closed the connection]
kchanqvq has quit [Ping timeout: 250 seconds]
triffid has joined #commonlisp
Oladon has joined #commonlisp
King_julian has quit [Read error: Connection reset by peer]
King_julian has joined #commonlisp
scymtym has quit [Ping timeout: 248 seconds]
scymtym has joined #commonlisp
puke has quit [Read error: Connection reset by peer]
puke has joined #commonlisp
scymtym has quit [Ping timeout: 256 seconds]
kchanqvq21 is now known as kchanqvq
vidak has joined #commonlisp
kchanqvq98 has joined #commonlisp
scymtym has joined #commonlisp
kchanqvq75 has joined #commonlisp
kchanqvq has quit [Ping timeout: 250 seconds]
kchanqvq75 is now known as kchanqvq
<beach>
equwal: Great!
kchanqvq98 has quit [Ping timeout: 250 seconds]
jonatack has quit [Ping timeout: 255 seconds]
danieli has quit [Quit: Ping timeout (120 seconds)]
danieli has joined #commonlisp
<beach>
pl: OK, but as soon as you have a pointer to some address, that pointer must be updated by the dynamic linker. Take this: https://plaster.tymoon.eu/v/7EA0WNVPM#4928 C program for instance. Each time it is run, a different value is printed.
saturn2 has quit [Ping timeout: 244 seconds]
saturn2 has joined #commonlisp
<beach>
So a Common Lisp implementation that is bootstrapped by writing an complete object graph to the executable file must then do whatever C does to add a linker thingy to each pointer so that the dynamic linker can fix up each pointer at startup time.
<beach>
s/an complete/a complete/
<pl>
or use base load address relative addressing and/or IP-relative addressing
<pl>
relocations are actually unrecommended these days
<beach>
You can't use PC-relative addressing for pointers.
sloanr` has joined #commonlisp
<pl>
umm, for this specific case? you can
<pl>
trivially so
sloanr has quit [Read error: Connection reset by peer]
<beach>
If you have a variable *FOO* and it contains a CONS cell, and you want to do (CAR *FOO*), you can't use PC-relative addressing for the *FOO* pointer. It has to be an absolute pointer, or you can use a base pointer, but then you must also execute an ADD instruction.
<pl>
beach: in fact, that code you linked compiles to RIP-relative addressing on AMD64
<beach>
pl: I don't see how the value of y can be RIP-relative.
<pl>
at least on my default compile options, if you don't have -fPIE, it will use text segment base address relocation from my understanding, but for over a decade everything has been -fPIE on my systems
<pl>
beach: size of .text is known, so you can calculate offset from IP to position in .bss
<beach>
pl: I don't think I am expressing myself very well.
<pl>
and I just linked listing showing that with -fPIE (default on multiple distributions) y is RIP-relative
<beach>
I can see how the address of x and the address of y can be RIP relative when referred to directly by code. But I can't see how the VALUE of y can be RIP relative. So the ADDRESS of x must change for every invocation.
<pl>
ah, value of y isn't, but address of x can be trivially calculated as RIP relative in this case
<beach>
That's my point. So if you think of X as containing a CONS cell created by a Common Lisp system, then the linker must determine its address.
<beach>
So I am wondering whether current Common Lisp systems use that technique. From what jackdaniel said, it is not the case for ECL which apparently creates the entire object graph at startup time. But what about the others.
<pl>
SBCL, with exception of a forgotten branch, uses fixed address
<beach>
It can't in the presence of ASLR.
<pl>
it overrides ASLR
<beach>
How does it do that?
attila_lendvai_ has joined #commonlisp
<pl>
ASLR triggers when you do not specify fixed-address mappings, i.e. if you tell the OS "I want a block of memory size X", without address
<pl>
on some OSes/kernel builds, there's a tunable that prevents fixed mappings
<pl>
I believe SBCL also set some set_personality calls to change ASLR behaviour on Linux in the past
attila_lendvai has quit [Read error: Connection reset by peer]
<beach>
I see. So if I understand you right, SBCL has its object graph in a file that it mmaps to a fixed location? Perhaps the file is the core file?
<pl>
yes
<beach>
Got it, thanks.
<pl>
well, core file is more like a custom ELF file (actually, SBCL's core file is ancestor to Mach-O executable used in MacOS)
<pl>
however, SBCL uses fixed base address to references when loading it
<beach>
The details are not important. What is important is that it doesn't deal with ASLR.
<beach>
Now, I personally think ASLR is a terrible idea that is meant to make an unsafe language a bit safer on an operating system using technology from the 1960s, and the cost of ASLR is that startup time is now must longer, given what the dynamic linker has to do, especially when the object graph is large.
kchanqvq58 has joined #commonlisp
<pl>
beach: honestly? on 64bit linux, there's very little work for linker with ASLR
<beach>
But gilberth thinks we (Common Lisp people) must adapt, so I am contemplating what technique to use.
<pl>
also, unless you operate on something like AS/400 or CHERI, yes ASLR can be important even in CL
msv has quit [Remote host closed the connection]
<beach>
pl: I can't see that. If I bootstrap SICL by writing the entire object graph to the executable file, including all the libraries etc., there will be millions of addresses that need to be fixed up at startup time.
kchanqvq74 has joined #commonlisp
<equwal>
to only support a statically linked os like plan9
<pl>
beach: see, in C with -fPIE, there are no relocations applied
<pl>
because the code is compiled to either use RIP-relative or PLT-relative addresses
<pl>
in fact, I'd often get compilation failure in the past because my OS rejected code with relocations
<beach>
I am afraid I don't understand how that can work.
kchanqvq has quit [Ping timeout: 250 seconds]
kchanqvq58 has quit [Ping timeout: 250 seconds]
kchanqvq76 has joined #commonlisp
<pl>
Every address that exists in on-disk file has well known offset by nature of C compilation. This means that if anything, there's work by compiler to generate IP-relative sequence to get the absolute address if needed
<pl>
Dynamic symbols referenced in ELF and resolved by system linker also have known *storage* location, meaning linker just fills in addresses into table
equwal has quit [Remote host closed the connection]
equwal has joined #commonlisp
kchanqvq74 has quit [Ping timeout: 250 seconds]
<pl>
Both PLT and .bss have all locations resolvable by simple IP relative addressing on major platforms, at least unless you have a biiiiig binary (over 2GB)
<beach>
If an "IP-relative sequence" is needed in order to do (CAR X) performance is dead.
shawnw has joined #commonlisp
<pl>
nah, you're forgetting that, essentially, C binaries do not have static objects like that
<beach>
But I am not talking about typical C programs. I am asking what a typical Common Lisp system is doing.
<pl>
and thus I am trying to setup a baseline to give differences
<pl>
IIRC, relocatable core SBCL branch just ran an O(n) process that updated all pointers upon load
<beach>
I have a pretty good idea what C is doing, especially given dynamically linked libraries, and that is too slow for a typical Common Lisp system.
<pl>
load-time costly, but trivial if you can distinguish pointers from other data
<pl>
also, RIP relative addressing is about one extra cycle latency on amd64
pabs3 has quit [Ping timeout: 245 seconds]
weeks has quit [Remote host closed the connection]
decweb has quit [Quit: Konversation terminated!]
pabs3 has joined #commonlisp
<pl>
beach: for a pure CL system with single address space, I'd use a "fixup raw pointers on load, inter module references use a table in the module" scheme
edgar-rft` has joined #commonlisp
<pl>
I assume the pointers are tagged enough in such situation and we don't have non lisp pointers mixed with lisp pointers
kchanqvq17 has joined #commonlisp
<beach>
I see. Thanks.
edgar-rft has quit [Ping timeout: 255 seconds]
<pl>
Decade-old hardware can easily "relocate" at 30 GB/s for a laptop
<pl>
For such a simple scheme of fixing up pointers to base load address that is
<beach>
OK.
kchanqvq76 has quit [Ping timeout: 250 seconds]
<beach>
gilberth: Do you know what CCL is doing in the presence of ASLR?
<beach>
bike: What is Clasp doing in the presence of ASLR?
wohonajax has quit [Quit: leaving]
scymtym has quit [Read error: Connection reset by peer]
scymtym has joined #commonlisp
msv has joined #commonlisp
sailorCa| has quit [Ping timeout: 258 seconds]
_whitelogger has joined #commonlisp
gjvc has joined #commonlisp
fgarcia has joined #commonlisp
semz has joined #commonlisp
Posterdati has joined #commonlisp
pve has joined #commonlisp
lusciouslover has quit [Read error: Connection reset by peer]
lusciouslover has joined #commonlisp
kchanqvq91 has joined #commonlisp
kchanqvq46 has joined #commonlisp
kchanqvq17 has quit [Ping timeout: 250 seconds]
kchanqvq91 has quit [Ping timeout: 250 seconds]
istewart has quit [Quit: Konversation terminated!]
admich1 has joined #commonlisp
Pixel_Outlaw has quit [Quit: Leaving]
apac has quit [Ping timeout: 256 seconds]
aeth_ has joined #commonlisp
aeth has quit [Quit: ...]
aeth has joined #commonlisp
aeth_ has quit [Quit: ...]
chomwitt has joined #commonlisp
kchanqvq25 has joined #commonlisp
kchanqvq46 has quit [Ping timeout: 250 seconds]
kchanqvq25 is now known as kchanqvq
rgherdt has joined #commonlisp
chomwitt has quit [Ping timeout: 248 seconds]
equwal has quit [Ping timeout: 260 seconds]
LainIwakura has joined #commonlisp
inline has quit [Quit: Leaving]
kchanqvq70 has joined #commonlisp
inline has joined #commonlisp
admich1 has quit [Ping timeout: 260 seconds]
admich1 has joined #commonlisp
kchanqvq has quit [Ping timeout: 250 seconds]
kchanqvq70 is now known as kchanqvq
admich1 has quit [Ping timeout: 258 seconds]
admich1 has joined #commonlisp
vidak has joined #commonlisp
Oladon has quit [Quit: Leaving.]
lcn_ has joined #commonlisp
vidak has quit [Quit: Konversation terminated!]
greaser|q has joined #commonlisp
greaser|q has quit [Changing host]
greaser|q is now known as GreaseMonkey
msv has quit [Read error: Connection reset by peer]
pkal has joined #commonlisp
treflip has joined #commonlisp
vidak has joined #commonlisp
admich1 has quit [Read error: Connection reset by peer]
lusciouslover has quit [Remote host closed the connection]
lusciouslover has joined #commonlisp
admich1 has quit [Read error: Connection reset by peer]
dra has joined #commonlisp
apac has joined #commonlisp
lusciouslover has quit [Remote host closed the connection]
anticomputer_ has joined #commonlisp
lusciouslover has joined #commonlisp
apac has quit [Read error: Connection reset by peer]
anticomputer has quit [Ping timeout: 272 seconds]
admich1 has joined #commonlisp
King_julian has quit [Read error: Connection reset by peer]
King_julian has joined #commonlisp
ewig has joined #commonlisp
bpanthi977 has quit [Ping timeout: 260 seconds]
<beach>
Maybe the simplest solution for dealing with ASLR is to create a bitmap with one bit for each 64-bot word in the heap, and a 1 where there is a pointer. It seems SBCL is around 35 Megabytes at startup, so there would be at most around 4 million pointers to update, probably a lot less. That shouldn't take very long at startup time.
zxcvz has joined #commonlisp
zxcvz has quit [Client Quit]
ec_ has joined #commonlisp
ec has quit [Ping timeout: 272 seconds]
JosephFerano has joined #commonlisp
JosephFerano has left #commonlisp [ERC 5.5.0.29.1 (IRC client for GNU Emacs 29.1.90)]
<pl>
beach: if the pointers are tagged (and thus precisely recognizable), you don't need a bitmap
<pl>
just a scan at load time to update them by offset
msv has joined #commonlisp
<pl>
beach: are we assuming this is Lisp operating in "foreign" environment (like Unix) or CLOSOS like?
Lord_of_Life has quit [Ping timeout: 256 seconds]
<beach>
pl: That won't work if there are specialized arrays of (unsigned-byte 64).
<pl>
beach: would they have a header in front declaring the type?
<pl>
and most importantly, size
<pl>
though generally I'm of a mind to separate pointer and guaranteed pointer-less data into separate pages
<beach>
Yes, but you would have to know where the header starts, so then you have to traverse the entire graph, dispatching on type. That's more complicated and that's why I suggested a bitmap.
<pl>
(where possible)
<beach>
Since I am talking ASLR, it is obviously Unix-likes.
<beach>
You wouldn't introduce such a stupid thing in CLOSOS.
<jackdaniel>
is it a dare? :)
<pl>
I would. I don't trust external environment and prefer defense in depth unless I have hardware-level capabilities, which are available at performance only on expensive IBM gear
decweb has joined #commonlisp
kchanqvq35 has joined #commonlisp
<pl>
anyway, let's say I have a lisp module/core file. I can traverse it *linearly*, in a loop that checks for tag/header of object, which allows me to skip over specialized arrays that can't contain pointers but can contain data that is mistaken for tagged pointer. I don't need a graph, I only need to fixup every pointer *on saving* to known base address (can be part of core/module file header).
<pl>
adjusting every pointer is then O(n=size) operation, possibly done fast enough I could trip a thermal warning on my CPU
kchanqvq has quit [Ping timeout: 250 seconds]
<pl>
if you absolutely need to go by graph, you can run collector on the GC roots and replace part of it with fixups
<pl>
since at load time you can easily guarantee none of the data or code is in use
yitzi has joined #commonlisp
rgherdt has quit [Ping timeout: 258 seconds]
shka has quit [Quit: Konversation terminated!]
shka has joined #commonlisp
rgherdt has joined #commonlisp
rgherdt has quit [Ping timeout: 260 seconds]
pestctrl has joined #commonlisp
<beach>
Like I said, it seems much easier to create a bitmap.
<beach>
pl: You seem to have many ideas about this. You should write a Common Lisp system and perhaps a Lisp OS.
<pl>
beach: I honestly want to. But by the time I have handled various obligations, I'm too out of energy
<beach>
I see.
<pl>
the above is literally from spending a lot of hours scrounged in various non-work moments thinking on many topics, including for a Lisp OS like system :/
<pl>
but musing while on walk with the dog != time to code anything and turn things into reality
<pl>
so it's possible I am missing something obvious, or some unknown unknowns, and my scheme would fail. I unfortunately don't have the resources to experiment with it properly :/
<beach>
I understand.
<pl>
the idea of code modules that are in form of <external symbol table><internal symbol table><code block that can freely optimize between functions within the block> for example came from discussing with gilberth about handling addressing beyond 4G code model on amd64
<pl>
but also from having played as a student with a contrib to SBCL that used partial core files to load programs faster than FASLs allowed
<pl>
beach: that said, I would be very interested in why you think linear scan *if every untagged data is preceeded by size-indicating header*, wouldn't work thus leading to graph-based solution. Even if I have low chances of using the knowledge in practice :)
yitzi has quit [Quit: yitzi]
<beach>
I can't guarantee that the memory is dense with Common Lisp objects, so I can't guarantee that a linear scan will always hit a Common Lisp object rather than some random junk.
kchanqvq35 has quit [Ping timeout: 250 seconds]
<pl>
beach: well, for the ASLR case, you only need to handle relocation when loading a dump from a previous instance (like SBCL core file, or some form of FASL)
<pl>
I *believe* one can guarantee the structure to not start with random junk, thus giving a stable interpretation unless the file is corrupted
<beach>
OK.
<pl>
then I am guaranteed that untagged value will be preceded by a tag describing that there's untagged data, and *how long the block is*
ec_ has quit [Ping timeout: 272 seconds]
treflip has quit [Ping timeout: 265 seconds]
ec_ has joined #commonlisp
<pl>
optimized loop will easily go 80GB/s on the hardware I am typing it on right now, assuming entire loaded object is essentially just CONS cells with pointers to other CONS cells (i.e. maximum pointerful case)
Lord_of_Life has joined #commonlisp
Lord_of_Life_ has joined #commonlisp
Lord_of_Life has quit [Ping timeout: 260 seconds]
Lord_of_Life_ has quit [Excess Flood]
mange has quit [Remote host closed the connection]
Lord_of_Life has joined #commonlisp
jA_cOp_ has joined #commonlisp
jA_cOp has quit [Ping timeout: 248 seconds]
yitzi has joined #commonlisp
yitzi has quit [Remote host closed the connection]
decweb has quit [Remote host closed the connection]
Thermoriax has joined #commonlisp
lusciouslover has joined #commonlisp
decweb has joined #commonlisp
admich1 has quit [Ping timeout: 250 seconds]
brokkoli_originl is now known as brokkoli_origin
viaken has joined #commonlisp
jdz_ is now known as jdz
bpanthi977 has joined #commonlisp
yitzi has joined #commonlisp
yitzi has quit [Client Quit]
admich1 has joined #commonlisp
chomwitt has joined #commonlisp
X-Scale has quit [Ping timeout: 248 seconds]
apac has joined #commonlisp
treflip has joined #commonlisp
yacin_ is now known as yacin\
yacin\ is now known as yacin
uhuh` has quit [Quit: ERC 5.5.0.29.1 (IRC client for GNU Emacs 29.4)]
shka has quit [Quit: Konversation terminated!]
shka has joined #commonlisp
bpanthi977 has quit [Ping timeout: 248 seconds]
treflip has quit [Read error: Connection reset by peer]
apac has quit [Quit: Konversation terminated!]
treflip has joined #commonlisp
bpanthi977 has joined #commonlisp
<gilberth>
beach: Good question. CCL disables ASLR. However, with regard to the heap CCL doesn't care at all at which address this heap is. If the address the heap was dumped with is available when loading it the again, the whole heap just gets relocated. However, it demands that below 4GB there is a jump table to subprimitives. Think of threaded code. And that it a most silly restriction.
<gilberth>
beach: On macos by default any address below 4GB is denied from being mapped by the binary including a zero "page" which just occupies this whole range. And that is tweaked.
bpanthi977 has quit [Ping timeout: 260 seconds]
<gilberth>
beach: So that might be the real issue and not ASLR as otherwise CCL would be fine with ASLR as far as I can tell.
X-Scale has joined #commonlisp
<gilberth>
I'm about to fix this jump table issue by having a NIL register. The longer I look at it the more I believe basing the AMD64 port on 32-bit x86 was an error. The AMD64 isn't as register starved and basing the port on PPC or ARM would have been nicer. I also have major headaches with the Lisp value stack and the C stack being separate. Signal handling isn't happy with it at all.
<gilberth>
Lisp value stack and regular CPU stack being separate is fine, however what confuses signal handling is that %rsp is switched back and forth. It would have been much saner if there would be another register set aside as the Lisp value stack pointer.
mal1 is now known as lieven
rgherdt has quit [Remote host closed the connection]
nullbath has joined #commonlisp
rgherdt has joined #commonlisp
decweb has quit [Quit: Konversation terminated!]
decweb has joined #commonlisp
bpanthi977 has joined #commonlisp
bpanthi977 has quit [Ping timeout: 260 seconds]
apac has joined #commonlisp
ewig has quit [Read error: Connection reset by peer]
rgherdt_ has joined #commonlisp
rgherdt has quit [Read error: Connection reset by peer]
bpanthi977 has joined #commonlisp
Pixel_Outlaw has joined #commonlisp
apac has quit [Ping timeout: 248 seconds]
gooba has quit [Remote host closed the connection]
apac has joined #commonlisp
gooba has joined #commonlisp
Everything has joined #commonlisp
pkal has quit [Ping timeout: 260 seconds]
gko` has quit [Remote host closed the connection]
gko has joined #commonlisp
yitzi has joined #commonlisp
gko has quit [Remote host closed the connection]
<pl>
gilberth: ... I sorta expected that there was, I can imagine switching *SS* back and forth back when using segmentation
decweb has quit [Quit: Konversation terminated!]
decweb has joined #commonlisp
admich1 has quit [Ping timeout: 258 seconds]
treflip has quit [Ping timeout: 260 seconds]
yitzi has quit [Quit: yitzi]
equwal has joined #commonlisp
Pixel_Outlaw has quit [Quit: Leaving]
<beach>
gilberth: Thanks. I am off for tonight, but I'll read up tomorrow morning.
admich1 has joined #commonlisp
bpanthi977 has quit [Ping timeout: 265 seconds]
bpanthi977 has joined #commonlisp
divlamir has quit [Read error: Connection reset by peer]
divlamir has joined #commonlisp
Everything has quit [Quit: leaving]
varjag has joined #commonlisp
bpanthi977 has quit [Ping timeout: 248 seconds]
leeb has joined #commonlisp
leeb_ has quit [Ping timeout: 260 seconds]
bpanthi977 has joined #commonlisp
olivial has joined #commonlisp
olivial_ has quit [Ping timeout: 250 seconds]
bpanthi977 has quit [Ping timeout: 260 seconds]
kchanqvq has joined #commonlisp
jon_atack has joined #commonlisp
Oladon has joined #commonlisp
jonatack has quit [Ping timeout: 265 seconds]
shka has quit [Quit: Konversation terminated!]
rgherdt_ has quit [Remote host closed the connection]
rgherdt_ has joined #commonlisp
shka has joined #commonlisp
alendvai__ has joined #commonlisp
attila_lendvai_ has quit [Ping timeout: 244 seconds]
pve has quit [Quit: leaving]
kchanqvq has quit [Ping timeout: 250 seconds]
bpanthi977 has joined #commonlisp
admich1 has quit [Remote host closed the connection]
admich1 has joined #commonlisp
lusciouslover has quit [Remote host closed the connection]
lusciouslover has joined #commonlisp
kchanqvq has joined #commonlisp
bpanthi977 has quit [Ping timeout: 260 seconds]
kchanqvq23 has joined #commonlisp
kchanqvq has quit [Ping timeout: 250 seconds]
bpanthi977 has joined #commonlisp
bpanthi977 has quit [Ping timeout: 258 seconds]
bpanthi977 has joined #commonlisp
lcn_ has quit [Quit: Leaving]
equwal has quit [Ping timeout: 260 seconds]
robin_ is now known as robin
Lycurgus has joined #commonlisp
kchanqvq23 has quit [Ping timeout: 250 seconds]
lcn_ has joined #commonlisp
admich1 has quit [Read error: Connection reset by peer]
bpanthi977 has quit [Ping timeout: 248 seconds]
alendvai__ has quit [Ping timeout: 248 seconds]
bpanthi977 has joined #commonlisp
admich1 has joined #commonlisp
lcn_ is now known as donlcn
bpanthi977 has quit [Remote host closed the connection]