Frostillicus has quit [Read error: Connection reset by peer]
Frostillicus has joined #ocaml
Frostillicus has quit [Read error: Connection reset by peer]
Tuplanolla has quit [Ping timeout: 240 seconds]
Frostillicus has joined #ocaml
Frostillicus has quit [Ping timeout: 260 seconds]
humasect has quit [Remote host closed the connection]
Frostillicus has joined #ocaml
humasect has joined #ocaml
amadaluzia has joined #ocaml
humasect has quit [Quit: Leaving...]
ygrek has joined #ocaml
ygrek has quit [Ping timeout: 244 seconds]
Frostillicus has quit [Ping timeout: 248 seconds]
Frostillicus has joined #ocaml
myrkraverk_ has joined #ocaml
Frostillicus has quit [Ping timeout: 276 seconds]
myrkraverk has quit [Ping timeout: 260 seconds]
agentcasey has joined #ocaml
agentcasey_ has quit [Ping timeout: 272 seconds]
agentcasey has quit [Remote host closed the connection]
agentcasey has joined #ocaml
myrkraverk has joined #ocaml
myrkraverk_ has quit [Ping timeout: 248 seconds]
casastorta has quit [Ping timeout: 260 seconds]
casastorta has joined #ocaml
Serpent7776 has joined #ocaml
Haudegen has joined #ocaml
Tuplanolla has joined #ocaml
Frostillicus has joined #ocaml
zor has joined #ocaml
Mister_Magister_ has joined #ocaml
Mister_Magister has quit [Ping timeout: 240 seconds]
Mister_Magister_ is now known as Mister_Magister
YuGiOhJCJ has joined #ocaml
bartholin has joined #ocaml
amadaluzia has quit [Ping timeout: 260 seconds]
<discocaml>
<drupyog> @octachron I thin I don't have the same notion of "reasonably easy to do" T_T
<discocaml>
<drupyog> @octachron I think I don't have the same notion of "reasonably easy to do" T_T
<discocaml>
<drupyog> I don't even understand how memprof events are triggered by normal allocations
<discocaml>
<octachron> I will try to have a deeper look later.
itszor has joined #ocaml
zor has quit [Ping timeout: 240 seconds]
<discocaml>
<drupyog> (At this point, I'm also open to an hack-ish solution that allows me to get the measurements for the paper 🥲
<discocaml>
<drupyog> (At this point, I'm also open to an hack-ish solution that allows me to get the measurements for the paper 🥲)
Frostillicus has quit [Read error: Connection reset by peer]
Frostillicus has joined #ocaml
YuGiOhJCJ has quit [Ping timeout: 244 seconds]
YuGiOhJCJ has joined #ocaml
hannes has joined #ocaml
amadaluzia has joined #ocaml
<discocaml>
<octachron> Before I meander between hacks, what do you want to measure precisely? The stack consumed in the main stack (aka the difference between the stack high and the sp pointer? Or the memory measured by stack? If it is the first option, is it fine if the hack is ruining time performance?
<discocaml>
<drupyog> Basically, the entier problem is: I have a function (which does use effects, notably concurrency), and I want to know its peak memory consumption, including stack
<discocaml>
<drupyog> everything idea is fair game
<discocaml>
<drupyog> everything is fair game
<discocaml>
<drupyog> I do not care about performance for this measurement
<discocaml>
<drupyog> I do not care about (time) performance for this measurement
<discocaml>
<octachron> So it is fine to instrument by hand?
<discocaml>
<drupyog> yes
<discocaml>
<drupyog> but I would like to be sure that it's the actual memory consumption, not some idealize version "I think it does this" 😉
<discocaml>
<contificate> maybe hook parts of runtime.s and traverse the fibers?
<discocaml>
<octachron> What seems doable to me in one day would be:
<discocaml>
<octachron> - Tracking the consumed stack space of all suspended stacks with instrumentation, and using a gc alarm to track the max of GC heap + suspended stack.
<discocaml>
<drupyog> The runtime seems to maintain a pool of fibers, I don't understand how to know which fibers are live. I don't quite understand how it even reclaims unreachable fibers
<discocaml>
<octachron> It doesn't, fibers are only freed after being run
<discocaml>
<octachron> (or discontinued)
<discocaml>
<drupyog> My initial attempt was using GC.alarm (before realising nothing related to fibers was available in stat), so the idea seems fine to me, but I don't know how to do the first part.
<discocaml>
<drupyog> Or I don't know what you mean by "instrumentation" in this case
<discocaml>
<drupyog> (My initial understanding, which was that fibers were on the heap, seems quite false, from the implementation I'm reading)
<discocaml>
<octachron> Rewrite `Effect e, k -> f k` into `Effect e, k -> bookkeep f k`
<discocaml>
<octachron> There are not on the OCaml GC-ed heap.
<discocaml>
<drupyog> @octachron and just add a counter incremented on recursive calls ? heh.
<discocaml>
<drupyog> I must admit I don't trust that to be even roughly indicative on the actual memory used, but I could try.
<discocaml>
<octachron> I was more thinking on using the fact that for suspended stack, you have the current stack pointer and the stack_high reified in the stack_info structure.
<discocaml>
<drupyog> aaah, ok
<discocaml>
<drupyog> so `bookkeep` is a call to a modified runtime that gives the current fiber high
<discocaml>
<drupyog> I like that a bit better
<discocaml>
<octachron> I think you don't even need a modified runtime.
<discocaml>
<drupyog> Do you have an idea how to access `stack_info` from `k` ?
<discocaml>
<octachron> aka a `take_cont_noexc` in `effect.ml`.
<discocaml>
<drupyog> I don't think I understand how to obtain the height via that
<discocaml>
<octachron> `sp - Stack_high` or the reverse?
<discocaml>
<contificate> would be reverse + width of stack_handler iirc
<discocaml>
<drupyog> (Did I already mentioned that I loathe C ? 🥲 )
<discocaml>
<contificate> hook it all in frida, then you can write JavaScript 🥴
<discocaml>
<octachron> Note that this sounds like a fun project, so I am planning to have a go at a hackish implementation this afternoon while waiting for opam for the 5.4.0~beta1 release.
<discocaml>
<drupyog> ok
<discocaml>
<contificate> what is it you're actually wanting to measure
<discocaml>
<drupyog> I'll let you do that rather than trying to wrap my head around crossing boundaries back and forth between caml and its runtime
<discocaml>
<contificate> I'd probably hook specific mallocs and just record the total extent of memory allocated for fibers overall
<discocaml>
<contificate> would need only determine if the call's ancestor is caml_alloc_stack - then I think they juse use realloc + free later (macros defined to use mmap but it's malloc by default)
<discocaml>
<drupyog> (I have a few variations of programs like that, with a various balance of stack and scheduler usage ....)
myrkraverk_ has joined #ocaml
<discocaml>
<contificate> turns out GC logs actually log fiber allocs
myrkraverk has quit [Ping timeout: 248 seconds]
<companion_cube>
You mean statmemprof?
<discocaml>
<contificate> well it's unclear what Drup wants to actually measure
<discocaml>
<contificate> who cares about Stack_high - sp, the fibers start off small and can be reallocated
<discocaml>
<contificate> it's like 32 words + the metadata
<discocaml>
<contificate> I'd say you ought to just measure the total max live allocations arising from mallocs/mmaps in fiber allocation code (which works around the stack_cache problem)
<discocaml>
<contificate> but whatever, I assume Drup has blocked me as he's just being having a personal correspondence with octrachron here
<discocaml>
<drupyog> Huh, I'm not quite sure you assume any hostility from me, that's not my intention. I have just no idea how to implement any of your suggestion to actually measure what I want (which is, as I said, peak memory, including the stack, of a function)
<discocaml>
<drupyog> Huh, I'm not quite sure you assume any hostility from me, that's not my intention. I have just no idea how to implement any of your suggestion to actually measure what I want (which is, as I said, peak memory, including the stack(s), of a function)
<discocaml>
<drupyog> given how the GC function, and the fact that it has a pool of fibers that it reuses, I don't see how measuing malloc/mmap is better than doing an external measurement of the OCaml program (and that's just extremely unprecise)
<discocaml>
<contificate> it's not of the OCaml program
<discocaml>
<contificate> you would determine if it's malloc'd from within the fiber alloc code
<discocaml>
<contificate> so you'd know, across the entire run of the program, what the entire malloc'd extent of fibers is
<discocaml>
<contificate> not even just the overall, but the max live allocations
myrkraverk_ is now known as myrkraverk
<discocaml>
<drupyog> Doesn't that only give you the total number of allocations ?
<discocaml>
<drupyog> Doesn't that only give you the total number of allocations, not the live amount ?
<discocaml>
<contificate> you'd record freeing, too - assuming it does just free them
<discocaml>
<contificate> then each alloc is summed against the current known allocs, then max is taken
<discocaml>
<drupyog> (in any case, @octachron has been successfully nerdsniped, so I will see what he cooks and do something else in the meantime :p)
<discocaml>
<contificate> alright, well I hope you get the measurements you need
YuGiOhJCJ has quit [Quit: YuGiOhJCJ]
Haudegen has quit [Quit: Bin weg.]
johnridesabike has joined #ocaml
Frostillicus has quit [Remote host closed the connection]
Haudegen has joined #ocaml
Frostillicus has joined #ocaml
tronexte has quit [Ping timeout: 244 seconds]
tronexte has joined #ocaml
Frostillicus has quit [Remote host closed the connection]
Frostillicus has joined #ocaml
<discocaml>
<contificate> can't lie, my switch of OCaml seems to keep fibers live for a very long time
<discocaml>
<contificate> maybe my demo program is broken, but it takes like _a lot_ of fiber allocations before some freeing happens
<discocaml>
<octachron> @contificate , freeing (in the sense of giving back malloced memory) only happen from stacks outside of the stack cache (with size greater than 512 words)
<discocaml>
<contificate> doesn't this mean you can just accumulate loads accidentally
<discocaml>
<contificate> I managed to get valgrind to report like 200 million bytes allocated at exit
<discocaml>
<contificate> pathological, perhaps
<discocaml>
<octachron> Yes, I am surprised by how optimistic is the stack cache.
Frostillicus has quit [Remote host closed the connection]
Frostillicus has joined #ocaml
<discocaml>
<drupyog> Nice @octachron ! What's the difference between the two ? It seems to expose an effect that returns the stack size in both cases ?
<discocaml>
<drupyog> (the last gist is incomplete)
<discocaml>
<octachron> Fixed. The first version uses an effect to capture the stack and introspect its size using the normal C FFI. This requires to do the wiring to do the accounting by hand.
LainIwakura has joined #ocaml
<discocaml>
<octachron> The second version adds a counter to the OCaml runtime that tracks the allocation/deallocation of stacks. The active version considers that the stacks parked in the cache are "morally freed" whereas the normal version only counts malloc allocation/deallocation.
<discocaml>
<drupyog> excelent
euphores has joined #ocaml
<discocaml>
<drupyog> (I'm impressed, it seems you have improved your runtime-fu significantly in the last few years :D)
<discocaml>
<octachron> Beware that the second version cannot have a precise view of consumed space stacks and only has the view of the stack size, but since the stack starts at 32 words and grow exponentially from there this gives at least an order of magnitude.
<discocaml>
<drupyog> Right, I'll give the first method a try first, to see if the wiring isn't too painful in my case (and because it allows people to reproduce the measures without an ocaml patch)
<discocaml>
<octachron> A possible obstacle for this method which is hidden between the lines of the example is that you cannot perform effects in Gc alarms.
<discocaml>
<drupyog> Ah, indeed, but I should be able to walk around that
Haudegen has quit [Quit: Bin weg.]
LainIwakura has quit [Ping timeout: 272 seconds]
Haudegen has joined #ocaml
dhil has joined #ocaml
Frostillicus has quit [Remote host closed the connection]
Frostillicus has joined #ocaml
humasect has joined #ocaml
LainIwakura has joined #ocaml
ridcully has quit [Quit: WeeChat 4.6.3]
ridcully has joined #ocaml
LainIwakura has quit [Quit: Client closed]
Frostillicus has quit [Ping timeout: 276 seconds]
Frostillicus has joined #ocaml
humasect has quit [Remote host closed the connection]
euphores has quit [Quit: Leaving.]
myrkraverk_ has joined #ocaml
myrkraverk has quit [Ping timeout: 248 seconds]
<discocaml>
<contificate> What's the state of type info when building the compiler? Is it still the case that it's like some partial support or something (like merlin kinda works but inconsistently)? I don't suppose you can yet do `dune build` to build `ocamlopt`?