<FromGitter>
<jots_twitter> I read that but I didn't see anything that helps. also, python is 10 times as fast as crystal as well. maybe not a big deal until dealing with multi GB files like i am right now...
<A124>
Well, Ruby has File.foreach to iterate over lines. Also why are you using \s and not \n?
<A124>
I'm trying to figure out stuff myself, I took a trip to the source, wrote stuff, then got stuck without documentation.
<FromGitter>
<jots_twitter> the split is a different thing than reading lines. i wanted to split each line into tokens on whitespace, but first i need to at least read in the lines at a reasonable rate... like at least as fast as ruby or python. need the speeeed...
<A124>
If you code python then python + pypy
<A124>
Wonder if the speed is code, caching or other reason though.
<FromGitter>
<jots_twitter> I think either the IO system is slow or I'm not using it right... all i'm trying to do is count lines.
<A124>
About the recursive hash, I have the problem when I want to create Hash to populate later, I cannot.
<A124>
It wants to know what types will be stored in it, and when I say hash, it turns into recursvie problem.
<crystal-gh>
[crystal] tbrand closed pull request #3538: `crystal build` without spepecified program files (master...feature-build-cmd) https://git.io/vX6Gp
<FromGitter>
<cjgajard> @jots_twitter if you do `time crystal file.cr --release` it need to parse, infer types, compile, do llvm optimizations, link and finally run, that will be slow.. but if you do `crystal build --release file.cr` and then `time ./file` the perfomance is just 1ms slower than `wc -l` (at least in my pc)
<FromGitter>
<jots_twitter> I am doing crystal build --release file.cr. how is mine so much slower? maybe something weird going on on my machine. will try to run it on another one.
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
soveran has joined #crystal-lang
soveran has joined #crystal-lang
soveran has quit [Changing host]
pawnbox has quit [Ping timeout: 258 seconds]
pawnbox has joined #crystal-lang
bjz has joined #crystal-lang
bjz has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
pawnbox has quit [Ping timeout: 260 seconds]
bjz has joined #crystal-lang
soveran has quit [Remote host closed the connection]
soveran has joined #crystal-lang
pawnbox has joined #crystal-lang
gloscombe has joined #crystal-lang
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
pawnbox has quit [Ping timeout: 246 seconds]
bjz_ has joined #crystal-lang
bjz has quit [Ping timeout: 265 seconds]
mark_66 has joined #crystal-lang
bjz_ has quit [Ping timeout: 250 seconds]
bjz has joined #crystal-lang
pawnbox has joined #crystal-lang
gloscombe has quit [Ping timeout: 240 seconds]
gloscombe has joined #crystal-lang
bjz has quit [Ping timeout: 248 seconds]
bjz_ has joined #crystal-lang
soveran has quit [Remote host closed the connection]
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
soveran has joined #crystal-lang
gloscombe has quit [Quit: Lost terminal]
pawnbox has quit [Ping timeout: 260 seconds]
soveran has quit [Remote host closed the connection]
soveran has joined #crystal-lang
soveran has joined #crystal-lang
soveran has quit [Changing host]
pawnbox has joined #crystal-lang
gloscombe has joined #crystal-lang
<FromGitter>
<luislavena> @jots_twitter there is already another *`wc` in Crystal* implementation, you might want to check how that works: https://github.com/sferik/wc.cr
<FromGitter>
<luislavena> you might only require change line 37 to use `=>` (so it becomes a hash) instead of a `NamedTuple`
soveran_ has joined #crystal-lang
Philpax_ has quit [Ping timeout: 248 seconds]
soveran has quit [Ping timeout: 256 seconds]
gloscombe has quit [Quit: Lost terminal]
Philpax_ has joined #crystal-lang
Philpax has joined #crystal-lang
Philpax_ has quit [Ping timeout: 260 seconds]
Philpax has quit [Ping timeout: 260 seconds]
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
pawnbox has quit [Ping timeout: 252 seconds]
pawnbox has joined #crystal-lang
Philpax has joined #crystal-lang
crystal-lang982 has joined #crystal-lang
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
crystal-lang982 has quit [Ping timeout: 260 seconds]
<crystal-gh>
[crystal] yxhuvud opened pull request #3541: Add argument to Array#flatten to specify amount of levels to flatten. (master...flatten) https://git.io/vXiF8
snsei has joined #crystal-lang
snsei has quit [Remote host closed the connection]
snsei has joined #crystal-lang
Philpax has quit [Ping timeout: 260 seconds]
pawnbox_ has joined #crystal-lang
pawnbox has quit [Ping timeout: 260 seconds]
<A124>
So, anyone any ideas how to contruct a recursive hash if one does not know the depth needed beforehand?
_jungh4ns has joined #crystal-lang
<A124>
On other hand I found out how memory efficient current stable Ruby is.
pduncan has joined #crystal-lang
<FromGitter>
<luislavena> @A124: as long you know the type of data, you can define the Hash without issues:
<crystal-gh>
[crystal] yxhuvud closed pull request #3541: Add argument to Array#flatten to specify amount of levels to flatten. (master...flatten) https://git.io/vXiF8
snsei has quit [Remote host closed the connection]
<FromGitter>
<bcardiff> @samueleaton you can play with pack/unpack to build new tuples
<FromGitter>
<samueleaton> Thanks. Very functional. I like it.
mark_66 has quit [Quit: Leaving.]
<A124>
@luislavena: Thank you, that is what I was looking for and missed. Yet I am lost on usage of the result. Do I need to declare new? Or I just do not get it and cannot seem to find the needed info to figure it out.
<A124>
Looking at making merkle tree. Each node having some local data and optionaly children of the same type/class.
snsei has quit [Remote host closed the connection]
snsei has joined #crystal-lang
<FromGitter>
<luislavena> @coderobe: check `Process.new` and `Process.run`, it lacks some documentation but is quite capable of run child processes
<coderobe>
Cool, i'll check it out
<FromGitter>
<luislavena> @A124: can't help with merkle tree, but I can recommend starting with the docs on Hash usage.
<A124>
@luislavena Merkle tree could just be recursive hash with one key => value being children. I already did that in Ruby. It is suprisingly efficient at memory, the speed when having large trees is not that great though.
<A124>
The problem is I am unable even to create a very basic thing. API says look at docs for basic usage, but the basic usage has nothing there.
<A124>
I would be very glad for an example of such Tree, as you gave the example. I'm lost. I just have no idea, trying random stuff at this point.
<coderobe>
i'm having a problem creating a string from a YAML.mapping: "no overload matches 'MyYamlMapping#to_yaml' with type YAML::Emitter", any ideas?
<A124>
It mentions JSON is example of recursive type, but cannot find anything helpful at the source either.
<A124>
How would one get single instance of the aliased type, instead of using array?
<A124>
The .new method does not work on the alias. And also apparently one cannot set a value of hash to an empty hash
<FromGitter>
<johnjansen> you can if you can type it i.e. `{} of String, String`
unshadow has joined #crystal-lang
<FromGitter>
<johnjansen> and you can `merge` and `merge!` hashes which might be helpful … perhaps
<A124>
But I am using custom type (with alias)
<FromGitter>
<johnjansen> without seeing that i personally cant offer anything other than hints
<unshadow>
Hi Guys, I'm using Crystal to build a load testing tool for testing Concurrent connections, It seems that I'm kinda hitting a wall when I want Crystal to generate above 20k fibers. I don't see any crashes or something, the machines that test are quite powerfull (36 Cores, 64Gb RAM) and don't seem to be impacted by the program. Is there any logical lock that I should know of ?
<unshadow>
Note: I have other programs that do generate more then 40k Concurrent Connections on the same machines, so the machines and Archetecture are not the blame
<A124>
@johnjansen Would you mind providing merkle tree example? { :name => "string", :data => "something", :children => {} }
<A124>
But the alias is (just a testing one) alias Tree = Hash(Symbol | String, String | Array(String) | UInt64 | Int32 | Tree)
mgarciaisaia has joined #crystal-lang
<A124>
I have no idea if there is even a single example on the internet. Github no luck, web search no luck.
<A124>
Thank you. Yes, that does help. I was asking earlier if I have to use a class. So there is no reasonable way to do that with just alias, I suppose. Is there?
<A124>
How does one create a single instance of aliased type, instead of array, like in example?
<crystal-gh>
[crystal] zatherz opened pull request #3542: JSON.mapping setter/getter generation control (master...feature/json_mapping_setter_getter) https://git.io/vXPwL
<BlaXpirit>
A124, i've been looking at your messages from the start but I still have no idea what you need
<RX14>
if you have an alias
<A124>
Merkle tree in a Hash structure.
<RX14>
you can just call .new on it
<A124>
Error in ./repl.cr:4: undefined method 'new' for Tree:Class
<FromGitter>
<johnjansen> @A124 i honestly think that you need to start with something you understand and then start trying things like alias types, you can substitiute Struct for Class in the hint i sent you … but see if you can make *something* work before getting tricky
<RX14>
if it's recursive it won't compile
<A124>
And that is what I need, yes. as I stated before, in Ruby I use { :name => "string", :data => "something", :children => {} }, and access it as root[:children]["name"] or root[:data]. Same aplies for nested levels.
<RX14>
uyou wouldn't want to do that in crystal
<A124>
Does 'won't compile' == 'not supported' ?
<RX14>
use a record
pduncan has joined #crystal-lang
snsei has quit [Ping timeout: 258 seconds]
<RX14>
A124, well what you want honestly should work
<RX14>
but it doesn't
<A124>
Alright, what is a record?
<RX14>
it's a macro to generate a class for holding data
<RX14>
you don't want to hold data that isn't key value data in Hash in crystal
<RX14>
you want to use something more concrete
<RX14>
so recird basically defines a class which holds the data
pawnbox_ has quit [Ping timeout: 260 seconds]
<FromGitter>
<sdogruyol> a record for example
<RX14>
and with getters and setters
<RX14>
thats a simplification but
<FromGitter>
<sdogruyol> @RX14 you mean a struct?
<RX14>
yes
<RX14>
but the difference doesn't matter for now
<A124>
What is the difference and between what and when it matters?
<FromGitter>
<johnjansen> @RX14 where abouts in the docs
<RX14>
under structs
<FromGitter>
<johnjansen> thanks ;-)
<unshadow>
sdogruyol : I remmber we had issues with it, (it's me bararchy) , I thought we worked around it by using Fibers\Channels no ?
<FromGitter>
<sdogruyol> @unshadow long time no see, well you can generate as much as fiber as you want but there's the event loop
<FromGitter>
<sdogruyol> your limit is the event loop
<FromGitter>
<johnjansen> ok who can remember how to dump the results of a macro
<FromGitter>
<johnjansen> some debug trick from what i remember
<FromGitter>
<sdogruyol> {{ %debug() }}
<RX14>
no %
<RX14>
just {% debug() %}
<RX14>
{{}} works too probably
<FromGitter>
<sdogruyol> ah yes thanks RX14
mgarciaisaia has left #crystal-lang [#crystal-lang]
<FromGitter>
<johnjansen> hmmm, i wanted to dump the result of your record example @RX14 for @A124 to no avail
<A124>
How do I dump the record result then?
<FromGitter>
<johnjansen> haha
<FromGitter>
<sdogruyol> lol
<unshadow>
sdogruyol: How much Concurrent Connection did you manage using Cryload ?
<FromGitter>
<sdogruyol> @unshadow something between 15k - 20k
<FromGitter>
<sdogruyol> i guess i hit the event loop limit
<FromGitter>
<sdogruyol> and for now crystal is single thread so
<FromGitter>
<johnjansen> anyone written a retry macro yet?
pduncan has quit [Ping timeout: 245 seconds]
pawnbox has joined #crystal-lang
<unshadow>
sdogruyol: So the event loop is responsible ? is there an hardcoded value ? or something we can play around with ? why do we hit a limit if no Hardware cap is reached ?
<FromGitter>
<sdogruyol> @unshadow i'm not sure if we can play with it. It's libevent
<A124>
RX14 So... is there a way to output resulting struct, also given 'you should prefer using structs for immutable data types' if it does pass value, if the tree is large does it pass whole tree, or just the single node? And if not structs, which shall I use, in case of mutable data (by design one has to have mutable, or construct from leaves up).
<RX14>
A124, well Array isn't a struct
<RX14>
so when you have children : Array(String) thats passing a pointer to the array
<RX14>
like usual
<RX14>
so no it's actually a small amount of memory
<FromGitter>
<johnjansen> @RX14 can you dump the output of that macro for him, i couldn’t make it work and im in the middle of retry :-)
<RX14>
you can't dump the macro outside the macro
<FromGitter>
<crisward> Has anyone wrote any tutorials on macros?
<RX14>
the docs are pretty good
<FromGitter>
<johnjansen> thanks @RX14
<FromGitter>
<sdogruyol> the docs are good but not so practical
<RX14>
i guess i just played around with it until it worked
<RX14>
A124, the struct will be
<FromGitter>
<crisward> I suppose I'm looking for examples of why / when to use them etc
<A124>
So I can just write the setters and getters myself. And compare bahaviour.
<A124>
I just wanted to see what the record macro actualy constructs.
<RX14>
you can sort of see it from the macro definition
<FromGitter>
<johnjansen> haha @rx14 you beat me to it
<RX14>
for every property you define in the record it genrates a getter
<RX14>
and then it generates an initialize like (@foo, @bar)
mgarciaisaia has joined #crystal-lang
<RX14>
to set the instance vars
<FromGitter>
<sdogruyol> has anyone tried to write a GC before?
<RX14>
then {{yield}} is so you can put a block on the macro
<RX14>
@sdogruyol I presume you mean in crystal
<RX14>
otherwise the answer is obvious :P
<FromGitter>
<sdogruyol> no
<FromGitter>
<sdogruyol> i asked it for everything
<FromGitter>
<sdogruyol> :P
mgarciaisaia has left #crystal-lang [#crystal-lang]
<FromGitter>
<sdogruyol> i'm interested in GCs and looking for where to start
<FromGitter>
<sdogruyol> and it'd be great to have someone experienced for ideas
soveran has joined #crystal-lang
<A124>
GCs are not a simple thing, so most people just implement the simple ones until there is need to implement the better ones.
<FromGitter>
<sdogruyol> yeah i assume that they're not simple .P
soveran has quit [Ping timeout: 240 seconds]
<A124>
Thanks RX14! So I can just copy paste macro and add a setter (does saying 'setter' suffice?, or should I do it differently?), and add default values for the initialize to simplify things. This is great.
<RX14>
you can add a = after the type in the record for a default
<RX14>
it's part of the language syntax
<A124>
Yeah, I know same as ruby, the question was about the setter.
<RX14>
ohh
<RX14>
well you can add a do
<RX14>
then write as if you're inside the class
<RX14>
or just write a vanilla struct from scratch
<A124>
Would just saying 'setter name' have problems with the copy thing?
<RX14>
honestly you don't want a setter in a record
<A124>
Then how do I modify it.
<RX14>
if you want a setter you want a class
<RX14>
records are meant to be immutable
<RX14>
because they're pass by value
<RX14>
so that changes inside a method don't change anything
<A124>
Well say it is a file system. If you modify a file, you want that leaf and all levels up changed.
<RX14>
yes, so use a class
<A124>
Yeah, I read that in API docs. Ok. Class for everything or just branches?
<A124>
Is there negative memory/performance implication in using class?
<RX14>
yes but not much
<RX14>
it's an extra malloc call
<A124>
Alright, thank you very much.
<A124>
... got one more unrelated question. How one would go about reading chunk of file and hashing that chunk with Digest?
<A124>
In Ruby, Go I read a part of the file, then give it to Digest, in Crystal I failed doing so.
<RX14>
yes
<RX14>
uhh
<RX14>
Digest::SHA1 is useless for this
<RX14>
let me just get some code
<A124>
Or Zlib, compress a chunk, to estimate compression.
<RX14>
basically
<A124>
I was offline so I had to rely on source, so I figured out how to use a lot, but got stuck, tried using MemoryIO, but had no idea how to use that as a "pipe.
<RX14>
you want to use OpenSSL::Digest
<RX14>
you don't want to use MemoryIO for this
<A124>
I wanted to read a chunk to bufer then do that, did not find how.
<FromGitter>
<luislavena> @jots_twitter as explained when linked, options is defined as a NamedTuple, not a Hash, a NamedTuple is immutable, so you can't `[]=` on it.
<FromGitter>
<luislavena> Change `{key: value}` syntax to use `=>`:
<FromGitter>
<luislavena> @jots_twitter no worries, I mentioned that when added the link, probably you missed.
<FromGitter>
<luislavena> @johnjansen either you can use `{{block.body}}` or `{{yield}}` to reference the block used as part of a macro invocation.
bjz has quit [Ping timeout: 245 seconds]
vifino has quit [Quit: Who turned this off?! D:<]
vifino has joined #crystal-lang
soveran has quit [Remote host closed the connection]
j2k has joined #crystal-lang
j2k has quit [Client Quit]
<FromGitter>
<jots_twitter> still messing around with crystal to get a decent wc working. running it on a 200MB file, it's 4 times as slow as perl and 20 times as slow as C. I really want to use crystal for slinging around text in multi GB files, hopefully I'm doing something wrong or there is a fix on the horizon. numbers: https://gist.github.com/anonymous/da6433854e67a1062ed36a668444dae1