corecheckno has quit [Remote host closed the connection]
rob_w has joined #picolisp
<abu[7]>
tankf33der, you are using (gc n1 n2) occasional
<abu[7]>
y, right?
<abu[7]>
This may not be necessary any longer, as Pil now adjusts the reserve size automatically
aw- has quit [Ping timeout: 252 seconds]
<beneroth>
abu[7], what does that mean, adjusting reserve size? according to which logic is the needed reserve estimated?
<beneroth>
I also use (gc n1) or (gc n1 n2) sometimes and I think I should use it more for better control of DB caching
<beneroth>
I'm all for sensible defaults and possibility of finer control
<abu[7]>
Yes, but I think now it is not so much needed any more :)
<abu[7]>
gc detects that it suddenly needed more heap
<beneroth>
how? when it is more often triggered within timeframe N or... ?
<abu[7]>
Not time
<abu[7]>
but how many cells are needed since last call
<beneroth>
my most significant use case for (gc) is data imports, when I build large object-structures (linked symbols) from data, pre-allocating memory makes a huge difference (maybe that changed now, I haven't yet tested with current versions of pil)
<beneroth>
I see, yeah that also gives an estimate of the changed cell consumption need
<abu[7]>
Yes, imports a typical case
<abu[7]>
Momenntt, demo
<beneroth>
I also use the NIL property in external symbols to cache results, where I use the wiping of the external symbol as cache invalidation
<beneroth>
kk
<beneroth>
thank you, we can continue later :)
<abu[7]>
~/pil21 ./pil +
<abu[7]>
: (bench (make (do 99999999 (link T)))) (heap)
<abu[7]>
2.849 sec
<abu[7]>
-> 1857
<abu[7]>
<abu[7]>
~/pil21 pil +
<abu[7]>
: (bench (make (do 99999999 (link T)))) (heap)
<abu[7]>
369.982 sec
<abu[7]>
-> 1527
<abu[7]>
<abu[7]>
First is new version
<beneroth>
impressive
<beneroth>
how does it compare to old version + manual (gc) call?
<abu[7]>
Very simple, It sees how many cells are needed, in compares to the GcCount global
<beneroth>
I mean.. its certainly a better universal allocator than before, question is if it is really that good that the manual control should be removed (= loss of programmer control)
<abu[7]>
GcCount is what is n2 in (gc n1 n2)
<abu[7]>
A one-liner in src/gc.l
<beneroth>
or lets ask the other way around.. how much extra work and maintenance effort is to keep the old interface while make the new behavior the default?
<abu[7]>
The old one works as before
<abu[7]>
Nothing changed
<abu[7]>
Just n2 is increased automatically
<beneroth>
I understand the new version does some more bookkeeping (on each cell allocation), though that cost is probably completely negligible compared to gc run + memory allocation?
<beneroth>
ah
<abu[7]>
No, no more book keeping
<abu[7]>
Just a single expression checking the free cells
<beneroth>
ah ok
<beneroth>
but it also must have a variable holding the old free cell value, no?
<beneroth>
but right, so only a check per gc run, not per cell use
<abu[7]>
Yes, GcCount
<abu[7]>
existed
<beneroth>
right
<beneroth>
so.. it's basically automation of the fine-tuning of n2 which previously had to be done by hand
<abu[7]>
exactly
<beneroth>
but n1 can be used as before, and n2 too (though its questionable if this can be done better manually then the automatic fine-tuning), do I understand now correctly?
<abu[7]>
Yes, perfect
<beneroth>
thanks! and thank for your patience, I just ask to grok ;)
<abu[7]>
n2 makes sense to have a big heap from the beginning o
<beneroth>
how does the new automatism shrink n2?
<beneroth>
(or GcCount)
<beneroth>
(being the same thing)
<abu[7]>
It never shrinks as before
<abu[7]>
only with (gc 0)
<beneroth>
ok
<beneroth>
is (gc 0 1024) a sensible call?
<beneroth>
aka "free whatever you can, give back to OS, but keep 1 GB of free cells available"
<abu[7]>
No, but (gc 2000 1024)
<abu[7]>
(gc 0) ignores n2 iirc
<beneroth>
ok
<abu[7]>
and shrinks once
<beneroth>
(gc 1024) doesn't shrink, only (gc 0), right
<beneroth>
?
<abu[7]>
T
<abu[7]>
I stumbled across this when working at the TPC-H stuff
<beneroth>
so "free whatever you can, give back to OS, but keep 1 GB of free cells available" is not possible (neither in the old nor new version), it would need to be "free whatever you can, give back to OS" using (gc 0) followed by a new reservation (gc 1024), ?
<beneroth>
ah nice
<abu[7]>
yes
<beneroth>
ok
<beneroth>
T
<abu[7]>
Just let it run, and do (gc 0) at proper points
<abu[7]>
no worry about the best n2 size
<beneroth>
so I guess for my use case of caching in NIL property of external symbols, possibly many of them, the sensible thing would be to make an initial call with an appropriate n1 and n2
<beneroth>
right?
<beneroth>
because every gc run does clean my NIL properties
<abu[7]>
It does not
<abu[7]>
only the next call to commit or rollback
<beneroth>
right, it only cleans the ones where the external symbol is not referenced (other than by *DB)
<abu[7]>
exactly
<beneroth>
I do cache method results this way
<abu[7]>
a good way
<beneroth>
and invalidate by (wipe) calls when another process modified the external symbol
<abu[7]>
I see
<beneroth>
this way I can be, that I have many external symbols with NIL properties which are not part of the current evaluation and thus not referenced in any variables, and they get cleaned either by explicit (wipe) (from current or other process) or by gc run, correct?
<beneroth>
it's quite a particular scenario
<beneroth>
I agree that in all other cases only (gc 'num) and (gc 0) needs be used
<abu[7]>
I found that 'enum' is very heartful to keep infos about externals
<beneroth>
I kinda found this NIL property mechanism more and more useful ;-)
<beneroth>
yeah
<abu[7]>
(gc 'num) does probably not make any difference
<beneroth>
picolisp (enum) is like a javascript array (which is not like an array at all in correct CS terminology)
<abu[7]>
space increases so fast
<abu[7]>
(automaticary
<abu[7]>
(automatically)
<beneroth>
well it could save some gc runs, when the cell tree is already very big, correct?
<beneroth>
but you're certainly right in the general case
<abu[7]>
it is always hard to predict
<beneroth>
yes
<beneroth>
T
<beneroth>
so the new recommendation is: don't do manual (gc) calls except maybe for (gc 0) to shrink memory usage after big processing, otherwise trust the automatism
<abu[7]>
T
<abu[7]>
let's do more benchmarks
<beneroth>
if you measure/profile using (rt) and know your load very well (and this estimate holds in the future), then you could manually fine-tune, but that is probably for very specific situations
<abu[7]>
T
<beneroth>
so better don't put that second part into the general recommendation, people who have such an issue will find this possibility with high likelihood themselves, as long as it is in the reference
<beneroth>
thanks for the explanation and this awesome improvement abu[7] :)
<beneroth>
<3
<abu[7]>
☺
<beneroth>
is the new version already published?
<abu[7]>
yes
<abu[7]>
on software-lab
<beneroth>
ok, I will try to upgrade then soonishly
<abu[7]>
and in tankf33der's repo probably
<beneroth>
tankf33der will surely test this very accurately :D
* beneroth
is excited for the results
<abu[7]>
:)
<beneroth>
very simple emergent feature, eh?
<beneroth>
(as so much in picolisp)
<abu[7]>
indeed
aw- has joined #picolisp
aw- has quit [Ping timeout: 252 seconds]
bjorkintosh has quit [Ping timeout: 276 seconds]
rob_w has quit [Remote host closed the connection]
aw- has joined #picolisp
aw- has quit [Ping timeout: 252 seconds]
bjorkintosh has joined #picolisp
bjorkintosh has quit [Read error: Connection reset by peer]