<Papierkorb>
jots_twitter, Hash basically works like this: Have a list of buckets, and each bucket can hold up to X elements. (X is this initial_capacity). At insert time, it is checked if the target bucket has enough space left, if yes, put it into it, else, create a new bucket
<FromGitter>
<jots_twitter> It made a difference: ⏎ ⏎ ``` code paste, see link``` ⏎ ⏎ So that's great. great numbers (to paraphrase someone). But still I don't know why creation so much slower than get. I would have thought about equal. [https://gitter.im/crystal-lang/crystal?at=5882a9a4300f220a66138080]
<Papierkorb>
I think I read it slightly wrong at first. Each bucket is in fact an Entry, which is a sort of double linked list
<Papierkorb>
jots_twitter, there's also the "benchmark" module in the stdlib
<Papierkorb>
jots_twitter, give it a try, you should be able to compare it to crystals Hash, the algorithm looks similar judging from the `private class Entry` :P
<FromGitter>
<jots_twitter> thanks for your help Papierkorb. more than double speedup with a single line change. vroom vroom.
<Papierkorb>
jots_twitter, also when benchmarking, make sure to use `--release` mode
soveran has joined #crystal-lang
<FromGitter>
<jots_twitter> always
<Papierkorb>
Actually, setting is faster for me than getting using your code and the initial_capacity mod
<Papierkorb>
set 33808944 per sec, get 247572060 per sec
<Papierkorb>
Oh no it's not
<Papierkorb>
That would've been weird
<Papierkorb>
Just overlooked some digits :P
<FromGitter>
<jots_twitter> 33_808_944 per sec, get 247_572_060 :-) still these numbers are yuge.
soveran has quit [Ping timeout: 258 seconds]
etrepat has quit [Ping timeout: 255 seconds]
bjz has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
bjz has joined #crystal-lang
bjz has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
bjz has joined #crystal-lang
ome has joined #crystal-lang
soveran has joined #crystal-lang
soveran has quit [Ping timeout: 240 seconds]
shelvacu has joined #crystal-lang
<shelvacu>
What's the rough equivalent to C's "#include<library/subfile.h>" when using a C library in crystal?
<shelvacu>
But that's passing flags to the linker, don't I also need some sort of include?
<FromGitter>
<bcardiff> The body of the lib section should be similar to the subfile.h probably. But there is no 1-1 correlation with a .h file to include.
<FromGitter>
<bcardiff> Yes. And you will need to either write them or generate it
<shelvacu>
Okay, I think I understand.
<FromGitter>
<bcardiff> I suggest write it initially and then if there is many functions generate it
ome has quit [Quit: Connection closed for inactivity]
fedruantine has quit [Read error: Connection reset by peer]
dhk has joined #crystal-lang
dhk has quit [Client Quit]
bjz has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
soveran has joined #crystal-lang
soveran has quit [Ping timeout: 255 seconds]
_whitelogger has joined #crystal-lang
matp has quit [Remote host closed the connection]
matp has joined #crystal-lang
triangles2 has quit [Ping timeout: 240 seconds]
triangles2 has joined #crystal-lang
bjmllr has quit [Ping timeout: 240 seconds]
bjmllr has joined #crystal-lang
soveran has joined #crystal-lang
lacour has quit [Quit: Leaving]
soveran has quit [Ping timeout: 240 seconds]
etrepat has joined #crystal-lang
etrepat has quit [Ping timeout: 264 seconds]
<FromGitter>
<yxhuvud> @papierkorb: rehash will update the bucket count, so a more correct statement is that everytime it reach 5X, it will iterate over all the elements, once. That should amortize to fast insertion in general. If that is not happening, then something is broken.
bjz has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
etrepat has joined #crystal-lang
etrepat has quit [Ping timeout: 264 seconds]
bjz has joined #crystal-lang
soveran has joined #crystal-lang
akwiatkowski has joined #crystal-lang
etrepat has joined #crystal-lang
bjz has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<RX14>
finalize will be run when it is garbage collected
<Yxhuvud>
It is not, but is garbage collection run before process exit? If it would, then it should have been called, no?
<RX14>
but it's very hard to force an object to be collected in artifical workloads
<RX14>
no
<RX14>
finalize is meant for resource collection
<RX14>
if tyhe process exist, the OS collects resources
<RX14>
so finalize doesn't need to be called
<Yxhuvud>
Assuming it is resources the OS automatically take back. If not, then it would need to be called.
<RX14>
well, thats an assumption crystal makes
<RX14>
and I think it's perfectly valid
<RX14>
it's bad to rely on finalize too much
<RX14>
it's basically meant for freeing memory allocated in C libs
<RX14>
IO::FileDescriptor uses it too as a sanity check
<RX14>
but you should never rely on that either
<Yxhuvud>
well, I'd assume it was meant for anything that should have the same lifetime as the object.
<Yxhuvud>
which of course includes memory, but could really include anything.
<RX14>
yes but the lifetime of the object is not guaranteed
<RX14>
at all
<RX14>
so explicit block-based `.open` stlye is always preferred
<Yxhuvud>
I wonder how other languages do here. I know python ends with a GC, but I have no idea about Ruby Java etc.
<RX14>
i'm basically repeating what @asterite has said, but I do agree with him on this
<FromGitter>
<crisward> Thanks for the clarification. The reason why I asked is, I was trying to work out if a spawn block inside a method would prevent a class instance from being garbage collected? I tried the above with and without spawn but didn't get finalize for either.
<RX14>
if the spawned fiber has a reference to the object
<RX14>
and if it's wrong, then is there really any harm?
<FromGitter>
<crisward> I'm using a very basic memory cache in class, using spawn as an equivalent to javascripts setTimeout to delete the cache after 60 seconds. Just worried is could cause a memory leak, as I'm unclear how the gc works.
<FromGitter>
<crisward> From the sound of things, I should be fine.
<RX14>
hmm
<RX14>
well if it has a reference to the memory cache
<RX14>
then the memory cache wont be collected
<RX14>
i think you'll have to use a weak reference or sth
<crystal-gh>
[crystal] RX14 opened pull request #3926: Add WeakRef to docs_main (master...patch-2) https://git.io/vMQfy
<Papierkorb>
RX14: Is it possible to wait on a Channel with a timeout?
<RX14>
not yet
<RX14>
it will if I ever finish my PR
<RX14>
there is a shard for it though
<Papierkorb>
Just asking, cause in that case, crisward could stop the deletion-fiber in the caches #finalize without making the code too awkward
<RX14>
well i'd still use the wekref
<RX14>
weakref*
<Papierkorb>
Yes of course
<RX14>
you dont need a finalize
<RX14>
in that case
<RX14>
because the fiber can see the GC status of the object without finalize
<RX14>
because the weakref value will be nil after GC
<Papierkorb>
Yes, my question aimed towards stopping the fiber not when it notices it, but to interrupt its sleep to make it go away not a minute later but 'now'
<RX14>
oh yes i guess thats an issue
<RX14>
just keep a fiber ref then
<RX14>
and call resume on it
<RX14>
actually that'll probably break the scheduler once the skeep fired
<RX14>
actually thats something that needs to be tried
<FromGitter>
<crisward> the only reference is path...
<RX14>
i dont think you do, though depending on how fast set_cache is called
<RX14>
you might be spawning way too amny fibers
<FromGitter>
<crisward> how many is too many? (if that's not a silly question)
<RX14>
30k / 60s
<RX14>
gives you the max rate per second you can spawn
<RX14>
so 500 calls per second
<FromGitter>
<crisward> I only call set_cache if the cache isn't already set. So I'd need 30k unique urls, and them all to be called in a second. Think I'm pretty safe there.
<FromGitter>
<crisward> 500 should be fine
<FromGitter>
<crisward> I'll make a note to make it scalable. What happens if this is exceeded, is an exception raised?
soveran has quit [Remote host closed the connection]
<RX14>
your program runs out of memory
<RX14>
and gets killed
soveran has joined #crystal-lang
<FromGitter>
<crisward> The cache is in memory anyway, so I'd probably eat a lot of memory there too. I could just check the size of the cache hash before setting and skip the cache when it's too big. This is just a quick a dirty solution to stop the db getting hit too much. Makes `ab` stats look way cooler without any db overhead.
<RX14>
well
<RX14>
uhh
<RX14>
it's the virtual memory limit it hits
<RX14>
at 256gb
<RX14>
so
<RX14>
idk why it hits at 256gb but it does for me
<FromGitter>
<crisward> Its high enough not to be a problem. Worth knowing though.
Philpax has joined #crystal-lang
soveran has quit [Remote host closed the connection]
etrepat has quit [Ping timeout: 264 seconds]
<RX14>
i'm stuck on a good way to compose HTTP::FormData with HTTP::Client
<RX14>
with HTTP::Server you just pass the request
<RX14>
and it can parse it
<RX14>
but there doesnt seem to be any easy way to hook up the code to the HTTP::Client