ChanServ changed the topic of #crystal-lang to: The Crystal programming language | http://crystal-lang.org | Crystal 0.31.1 | Fund Crystal's development: http://is.gd/X7PRtI | GH: https://github.com/crystal-lang/crystal | Docs: http://crystal-lang.org/docs/ | API: http://crystal-lang.org/api/ | Gitter: https://gitter.im/crystal-lang/crystal
<hightower4> Blacksmoke16 that event magic you did is just brilliant (almost done with incorporating/documenting the event model)
<hightower4> (I didn't have success with AnyEvent though (didn't manage to convert to module as it still didn't want to play with struct, nor convert to classes since then I got into issue of missing initialize()s which struct automatically provides), but that's of lesser importance for now)
<FromGitter> <Blacksmoke16> Just add the initializers
<FromGitter> <Blacksmoke16> The record macro defines one, not struct itself
<hightower4> ah cool, yeah, still I don't have a clear picture in mind how I'd do it.. (I wanted to generate initializes, rather than write them by hand for each class)
<FromGitter> <Blacksmoke16> Generate how?
<FromGitter> <Blacksmoke16> Could copy the record macro and make it define a class instead, but probably not worth it
<hightower4> I thought I could somehow find all the custom properties on the class, and generate from that
<hightower4> but yeah, that's not a bad idea, the elegance of how record is defined is great
<FromGitter> <Blacksmoke16> Ivars are only available within a method
HumanG33k has quit [Ping timeout: 265 seconds]
alexherbo2 has quit [Quit: Ping timeout (120 seconds)]
alexherbo2 has joined #crystal-lang
HumanG33k has joined #crystal-lang
Human_G33k has joined #crystal-lang
HumanG33k has quit [Read error: Connection reset by peer]
<FromGitter> <ImAHopelessDev_gitlab> ```code paste, see link``` ⏎ ⏎ first one to spot the error wins [https://gitter.im/crystal-lang/crystal?at=5db24e4d14d55a37856f2677]
<FromGitter> <Blacksmoke16> no colon after `as`
<FromGitter> <ImAHopelessDev_gitlab> :D
<FromGitter> <ImAHopelessDev_gitlab> You win!
<FromGitter> <ImAHopelessDev_gitlab> Runes.txt (https://files.gitter.im/crystal-lang/crystal/ckj9/Runes.txt)
<FromGitter> <ImAHopelessDev_gitlab> https://limonte.github.io/csv-viewer-online/ use to view. I'm doing rune words, that csv is how diablo 2 stored their rune word data
<FromGitter> <ImAHopelessDev_gitlab> right now, only got 1 rune word added and working on server. which is a good first step :)
hightower4 has quit [Read error: Connection reset by peer]
<FromGitter> <ImAHopelessDev_gitlab> with that said, `each_char` and `each` have been a godsend for socket / rune checking. sockets are stored in mysql as "000000" 0 is filler, 1 is socket, 2 is a linked socket (skill gem could be inserted in this as well), 3 is a rune word, and 4 is a gem. when inserted, the item's socket_data field gets updated. example: "0:25,1:26". socket position 0 and 1 are runes. sockets value is now
<FromGitter> ... "330000". ⏎ then, if a rune word needs to be checked, i can utilize `socket_data` and match it with the `sockets` to see what's inside. then, i can utilize the power of `.select` to obtain the possible rune words for that item type, and then call another `.select` after i extract the rune word string to validate if it's an actu ... [https://gitter.im/crystal-lang/crystal?at=5db25a939c398215097d4b3d]
f1reflyylmao has joined #crystal-lang
f1refly has quit [Ping timeout: 246 seconds]
teardown has joined #crystal-lang
<FromGitter> <christopherzimmerman> I'm running into a bottleneck in my code that I can't seem to fix. When I benchmark the `times` function, it runs quite fast for large numbers: ⏎ ⏎ ```times 100_000 increment 2 64.56k ( 15.49µs) (± 0.62%) 0.0B/op fastest``` ⏎ ⏎ Is the compiler unrolling the first loop but not the second? There has to be some major optimization going on that I am missing.
alexherbo2 has quit [Quit: Ping timeout (120 seconds)]
alexherbo2 has joined #crystal-lang
<FromGitter> <absolutejam_twitter> @hightower4 there's a data classes shard which does similar thinb as the record macro for classes
ht_ has joined #crystal-lang
<FromGitter> <Daniel-Worrall> I imagine the compiler can optimise single step loops a lot better than it can 2 step
<FromGitter> <Daniel-Worrall> The bytecode would come out a lot simpler
rohitpaulk has joined #crystal-lang
rohitpaulk has quit [Remote host closed the connection]
absolutejam2 has joined #crystal-lang
ht_ has quit [Quit: ht_]
absolutejam2 has quit [Ping timeout: 252 seconds]
absolutejam2 has joined #crystal-lang
dwdv has joined #crystal-lang
absolutejam2 has quit [Ping timeout: 268 seconds]
absolutejam2 has joined #crystal-lang
hightower4 has joined #crystal-lang
alexherbo21 has joined #crystal-lang
alexherbo2 has quit [Ping timeout: 240 seconds]
alexherbo216 has joined #crystal-lang
alexherbo21 has quit [Ping timeout: 240 seconds]
alexherbo216 has quit [Ping timeout: 265 seconds]
alexherbo2162 has joined #crystal-lang
<FromGitter> <vlazar> @christopherzimmerman I don't see the difference you shown. Do you mind posting you benchmark? I've tried benchmarking `i += 1` and `i += 2` cases and got the second one 2 times faster.
alexherbo21628 has joined #crystal-lang
alexherbo2162 has quit [Read error: Connection reset by peer]
alexherbo21628 has quit [Quit: The Lounge - https://thelounge.chat]
alexherbo2 has joined #crystal-lang
<FromGitter> <Blacksmoke16> @christopherzimmerman https://crystal-lang.org/api/0.31.1/Number.html#step(*,to=nil,by=1,&block)-instance-method maybe use this instead
duane has joined #crystal-lang
absolutejam2 has quit [Ping timeout: 245 seconds]
absolutejam2 has joined #crystal-lang
duane has quit [Ping timeout: 264 seconds]
alexherbo2 has quit [Ping timeout: 246 seconds]
alexherbo2 has joined #crystal-lang
dannyAAM has quit [Ping timeout: 245 seconds]
dannyAAM has joined #crystal-lang
alexherbo23 has joined #crystal-lang
alexherbo2 has quit [Ping timeout: 252 seconds]
alexherbo23 has quit [Ping timeout: 268 seconds]
<FromGitter> <greenbigfrog> Anyone able to give me a small example how to store arrays with kemal session?
alexherbo23 has joined #crystal-lang
alexherbo237 has joined #crystal-lang
alexherbo23 has quit [Ping timeout: 252 seconds]
duane has joined #crystal-lang
<FromGitter> <Blacksmoke16> can you stringify it?
alexherbo237 has quit [Quit: Ping timeout (120 seconds)]
alexherbo237 has joined #crystal-lang
<FromGitter> <christopherzimmerman> @Blacksmoke16 @vlazar here are my timings (https://gist.github.com/christopherzimmerman/c34968f4c5c91124a55a007885d4d5aa)
<FromGitter> <christopherzimmerman> I tried the step approach and it works fine when I pass literals, but I need to be able to use the instance variables. If it makes a difference, the instance variables will never change once the class is instantiated so maybe there is something I can do there.
<FromGitter> <christopherzimmerman> I understand why using variables would be slower than literals, but it baffles me that its 17000 times slower.
duane has quit [Ping timeout: 265 seconds]
<FromGitter> <tenebrousedge> @christopherzimmerman is this a pretty consequential bottleneck for you?
Yxhuvud has quit [Quit: No Ping reply in 180 seconds.]
<FromGitter> <christopherzimmerman> Yea it's pretty massive. I need to be able to support strided containers and I can't have them be thousands of times slower to access
<FromGitter> <kinxer> @christopherzimmerman An interesting comparison: https://play.crystal-lang.org/#/r/7vy9
<FromGitter> <tenebrousedge> @kinxer using play is not going to get an accurate benchmark
<FromGitter> <kinxer> I'm just using it to share the code.
<FromGitter> <kinxer> I compiled it on my machine, using `--release` (after getting an even worse comparison of ~20,000x from the code he had).
<FromGitter> <kinxer> Also, I've never even been able to make play run a benchmark.
<FromGitter> <tenebrousedge> oh, it does it, but it has to be a pretty simple benchmark
<FromGitter> <asterite> Yeah, for me all of the timings are similar. Did you compile/run with `--release`? The instance variable is read once so it shouldn't make a difference
<FromGitter> <christopherzimmerman> @asterite are you talking about my original timings or the ones just posted?
<FromGitter> <asterite> I'm talking about the benchmarks and the timings I get
<FromGitter> <kinxer> Both sets of benchmarks?
<FromGitter> <kinxer> (That is, `i` vs `2*i`.)
<FromGitter> <asterite> Here are my results: https://gist.github.com/asterite/22b1bcadd58575c8962e16404a2a230d
<FromGitter> <Blacksmoke16> also what OS @christopherzimmerman ?
<FromGitter> <asterite> Oh, with `i` it's very different
<FromGitter> <asterite> but all these benchmarks are flawed because you don't do anything with the result
<FromGitter> <asterite> so the compiler (in this case LLVM) is free to discard the results
<FromGitter> <asterite> more so with `i` than `i*2`
<FromGitter> <absolutejam_gitlab> woah, I didn't know Lucky used Crystal to render the HTML
<FromGitter> <tenebrousedge> personally I have a hard time distinguishing microseconds from nanoseconds under the best of circumstances
<FromGitter> <absolutejam_gitlab> That was badly worded
<FromGitter> <absolutejam_gitlab> I really want to use Crystal for my web app but there's not a solid GraphQL implementation
<FromGitter> <asterite> Another thing
<FromGitter> <asterite> when you pass literals the compiler is able to compute the result right way (again, this is done by LLVM)
<FromGitter> <asterite> with a local variable that doesn't change LLVM should be able to do the same
<FromGitter> <asterite> However, each block given to benchmark will form a closure over local variables at the top-level, and so this means the variables will be allocated on the heap and read from there
<FromGitter> <asterite> that's why the literal works so much faster
<FromGitter> <asterite> I'll try to move the var inside the block and see what happens
<FromGitter> <asterite> but I can't imagine `step` being a bottleneck in a program
<FromGitter> <kinxer> Yeah, using an outside variable is an even closer comparison (again, using play just to share code): https://play.crystal-lang.org/#/r/7vyh
<FromGitter> <asterite> same goes with `f`, which is closured so it will affect the benchmark, more so with these things that usually take nanoseconds to complete, so accessing the heap memory might be slower than that
<FromGitter> <asterite> This is a slightly better benchmark: https://gist.github.com/asterite/caf1d821ab23a4f6195ec39135b67f4e
<FromGitter> <asterite> Here there are no closured variables, and I also used `struct` for `Foo` since I assume it's immutable. All different approaches result in the same performance
<FromGitter> <tenebrousedge> nice
<FromGitter> <kinxer> And you used `&+=` as a much faster (non-overflowing?) operation?
<FromGitter> <asterite> I used it because otherwise it will overflow and stop the benchmark from running
<FromGitter> <christopherzimmerman> So if I'm understanding, my code wasn't necessary running slower, the benchmark was just bad?
<FromGitter> <kinxer> Right, but my other point was that it was faster than the `*=` I used above.
<FromGitter> <asterite> I think the `benchmark` module is in a way a bit flawed because results might change depending on closured variables, but if that's affecting the benchmark it means the benchmarked thing is probably super fast (like in this case, 1.2ns)
<FromGitter> <asterite> @christopherzimmerman kind of. Some blocks in the benchmark closured some variables and that happens to be slower than incrementing numbers
<FromGitter> <tenebrousedge> real-world performance measures are also usually more useful than artificial benchmarks
duane has joined #crystal-lang
<FromGitter> <christopherzimmerman> Really appreciate the help, thanks all!
absolutejam2 has quit [Ping timeout: 265 seconds]
<FromGitter> <Blacksmoke16> is there a crystal equivilent to https://www.php.net/manual/en/function.memory-get-peak-usage.php ?
<FromGitter> <Blacksmoke16> or is there a better way to test memory usage of a set of code?
<FromGitter> <Blacksmoke16> `GC.stats` maybe?
ht_ has joined #crystal-lang
<FromGitter> <Blacksmoke16> https://play.crystal-lang.org/#/r/7vz7 seems to work? :shrug:
<FromGitter> <tenebrousedge> 👍
Yxhuvud has joined #crystal-lang
alexherbo2377 has joined #crystal-lang
alexherbo237 has quit [Ping timeout: 265 seconds]
<FromGitter> <wontruefree> Are there any docs on crystals arithmetic order of operations?
alexherbo2377 has quit [Ping timeout: 268 seconds]
alexherbo2377 has joined #crystal-lang
alexherbo2377 has quit [Ping timeout: 240 seconds]
<FromGitter> <greenbigfrog> @Blacksmoke16 It's simply a array of JSONable objects ⏎ (worked around it for now by simply storing it seperately in redis, but I'd actually like it to expire like a kemal session)
alexherbo2377 has joined #crystal-lang
<FromGitter> <christopherzimmerman> @asterite I retried the benchmark you posted with `Foo` as a class instead of a struct and got this: https://gist.github.com/christopherzimmerman/c9cfba7cf4d3782494f0453444e82213. Is it because the compiler can optimize more if it knows `stride` and `size` can't change?
<FromGitter> <christopherzimmerman> Unfortunately I can't have my class as a struct since I need to be able to pass it by reference, but it seems that is where all the performance gain is.
<FromGitter> <jwoertink> If I make a file that does `raise "test"`, run it, then do `echo $?` I'll get 1 as a status code
<FromGitter> <jwoertink> assuming that file is `test.cr`, I have this code
<FromGitter> <jwoertink> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5db33b3bf26ea4729df56ab8]
<FromGitter> <jwoertink> If I run this file, my exit status is 0
<FromGitter> <jwoertink> Is there a way to get that proper status?
dannyAAM has quit [Quit: znc.saru.moe : ZNC 1.6.2 - http://znc.in]
dannyAAM has joined #crystal-lang
<FromGitter> <j8r> @jwoertink that's `exit $?`, not `echo $?`
<FromGitter> <jwoertink> I meant `echo $?`
<FromGitter> <jwoertink> does `exit $?` work in crystal?
<FromGitter> <jwoertink> I guess let me clarify what I'm doing here
<FromGitter> <jwoertink> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5db340fd3d669b28a0d7a402]
<FromGitter> <jwoertink> I want that second one to return 1, but I'm not sure what I'm missing to do that
<FromGitter> <j8r> sorry I thought it was in a script
alexherbo2377 has quit [Quit: Ping timeout (120 seconds)]
alexherbo2377 has joined #crystal-lang
<FromGitter> <jwoertink> no worries. I figured I should clarify what I was trying to do
<FromGitter> <j8r> I got `256` for return code
<FromGitter> <j8r> I thought it was due to the subshell, apparently not
<FromGitter> <j8r> `shell: false` returns the same code
<FromGitter> <j8r> That's n issue in crystal
<FromGitter> <j8r> `crystal eval "exit 256"` return `0`
<FromGitter> <jwoertink> oh
<FromGitter> <jwoertink> weird. Ok, I guess I'll open an issue
<FromGitter> <j8r> 👍
<FromGitter> <jwoertink> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5db344813d669b28a0d7bf7d]
<FromGitter> <jwoertink> lol
<FromGitter> <j8r> yep, `LibC.exit 256` too :/
<FromGitter> <jwoertink> maybe 256 isn't a valid exit code
<FromGitter> <j8r> it should be
<FromGitter> <jwoertink> 255 is ok
<FromGitter> <jwoertink> What is that, Unit8?
<FromGitter> <ilanpillemer> Is there a Queue class of some kind.. or idiomatic way to do queues in Crystal? (Prioritised/Queues or Heaps as well)
<FromGitter> <j8r> ho yes
<FromGitter> <bararchy> On POSIX systems the standard exit code is 0 for success and any number from 1 to 255 for anything else.
<FromGitter> <j8r> right, 256 is out of range
<FromGitter> <jwoertink> ok, that makes sense
<FromGitter> <j8r> so why 256...
<FromGitter> <jwoertink> but would my original issue still be valid?
duane has quit [Ping timeout: 276 seconds]
<FromGitter> <tenebrousedge> @ilanpillemer something like this (https://crystal-lang.org/api/0.19.2/Deque.html) ?
<FromGitter> <j8r> even `Process.run("exit 1", shell: true)` return `256`
<FromGitter> <j8r> I got an exit status of `512` lol
<FromGitter> <j8r> `Process.run("ls", {"-P"})`
<FromGitter> <jwoertink> lol
<FromGitter> <ilanpillemer> Yup. I searched for Queue!
<FromGitter> <j8r> there is an issue here
<FromGitter> <jwoertink> I don't really understand what the issue is. I would think if errors only go to 255, then we should never see 256 returned, right?
<FromGitter> <j8r> yes
<FromGitter> <j8r> The implementation uses `Int32`, because others OS other than POSIX may support higher exit status
<FromGitter> <j8r> I guess
<FromGitter> <jwoertink> ok, I'll open an issue
gangstacat has quit [Ping timeout: 240 seconds]
alexherbo2377 is now known as alex```
gangstacat has joined #crystal-lang
<FromGitter> <ilanpillemer> no recursive structs!
<FromGitter> <Blacksmoke16> there are two methods
<FromGitter> <Blacksmoke16> not `exit_status`
<FromGitter> <Blacksmoke16> @jwoertink
<FromGitter> <jwoertink> oh?
<FromGitter> <jwoertink> checking
<FromGitter> <tenebrousedge> @ilanpillemer recursive structs (https://github.com/crystal-lang/crystal/blob/master/src/json/any.cr#L20)
duane has joined #crystal-lang
dostoyevsky has quit [Quit: leaving]
<FromGitter> <jwoertink> oh! @Blacksmoke16 yeah I think that's what I needed
dostoyevsky has joined #crystal-lang
<FromGitter> <j8r> @tenebrousedge not really
<FromGitter> <j8r> recursive structs means a struct which has an ivar of itself
<FromGitter> <Blacksmoke16> i remember there being confusion between them before, but idk much about the implementation or what the diff is
<FromGitter> <j8r> `Process::Status` is certainly confusing
<FromGitter> <j8r> no return types for methods doesn't help
<FromGitter> <ilanpillemer> @tenebrousedge ⏎ ⏎ ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5db35a68e469ef4358652fa3]
<FromGitter> <tenebrousedge> See how `JSON::Any` works
<FromGitter> <ilanpillemer> I don’t think Json can point at itself down the chain
<FromGitter> <ilanpillemer> can it?
<FromGitter> <j8r> @tenebrousedge `JSON::Any::Type` doesn't point to `JSON::Any`
<FromGitter> <j8r> so not a recursive struct of itself
<FromGitter> <tenebrousedge> that's nice
<FromGitter> <ilanpillemer> yes.. so its not really recursive
<FromGitter> <ilanpillemer> its pseudo recursive
<FromGitter> <ilanpillemer> like a pig in sheeps clothing
<FromGitter> <ilanpillemer> 🐽🐑
baweaver is now known as lemur
lemur is now known as baweaver
<FromGitter> <christopherzimmerman> Has anyone looked into putting together a jupyter kernel for Crystal?
<FromGitter> <Blacksmoke16> @tenebrousedge you might have some ideas on this
<FromGitter> <tenebrousedge> D:
<FromGitter> <Blacksmoke16> trying to determine the best way to handle lazily querying a large number of DB records
<FromGitter> <Blacksmoke16> i.e. like instead of doing `SomeModel.all` and loading them all in
<FromGitter> <Blacksmoke16> ideally iterate over them one at a time so that they aren't all in memory at once
<FromGitter> <ilanpillemer> like an interator?
<FromGitter> <Blacksmoke16> exactly yea
<FromGitter> <tenebrousedge> :D
<FromGitter> <ilanpillemer> stream
<FromGitter> <Blacksmoke16> but after the record has been yielded, GC it
ht_ has quit [Remote host closed the connection]
<FromGitter> <ilanpillemer> can the DB do cursors?
<FromGitter> <Blacksmoke16> like a crystal version of that
duane has quit [Ping timeout: 246 seconds]
<FromGitter> <Blacksmoke16> there is http://crystal-lang.github.io/crystal-db/api/0.7.0/DB/QueryMethods.html#query_each(query,*args_,args:Array?=nil,&block)-instance-method
<FromGitter> <Blacksmoke16> which seems like is what i want? it helps on the memory front but not as low as i thought it would be when its done
<FromGitter> <tenebrousedge> cursor sounds like your boy there
<FromGitter> <tenebrousedge> if you want to do it low-level
<FromGitter> <tenebrousedge> the way `query_each` is done, it seems like it fetches the whole result, and then yields each element
<FromGitter> <tenebrousedge> so you minimize db roundtrips at the expense of memory
<FromGitter> <Blacksmoke16> i think i could add something to granite that wraps that
<FromGitter> <tenebrousedge> if you really want to minimize memory, you probably need to use cursors
<FromGitter> <kaukas_gitlab> Hi! Upgrading Crystal one of my specs broke. Turns out in the latest version `[false, true, true, true].first(3)` results in `[false, false, false]`. Why?...
<FromGitter> <Blacksmoke16> like either way the data is going to have to be selected from the db yea?
<FromGitter> <tenebrousedge> you can do it piecemeal, but you're going to pay some kind of performance penalty either way
<FromGitter> <Blacksmoke16> so ideal flow would be like `select data from db, for each RS new up a model obj and yield it, gc that object, repeat until no data left`
<FromGitter> <Blacksmoke16> then you're only loading the one class obj into memory at a time?
<FromGitter> <Blacksmoke16> which would be like 50% reduction in memory yea?
<FromGitter> <tenebrousedge> I think so
<FromGitter> <tenebrousedge> I mean, no idea about the memory reduction
<FromGitter> <Blacksmoke16> ill have to test it :P
<FromGitter> <kaukas_gitlab> Actually, `Array(Bool)#first(n)` seems to always return `false` `n` times.
<FromGitter> <Blacksmoke16> ill keep you posted
<FromGitter> <tenebrousedge> but if performance was really important then I would not rely on the ORM, and write a straight SQL query
<FromGitter> <tenebrousedge> @kaukas_gitlab that doesn't sound good
<FromGitter> <ilanpillemer> man day 11 2016
<FromGitter> <Blacksmoke16> `pp [false, true, true, true][0...3]` happens too
<FromGitter> <Blacksmoke16> prob is a bug
<FromGitter> <kaukas_gitlab> Wow
<FromGitter> <ilanpillemer> has had been in its clutches for ages
<FromGitter> <Blacksmoke16> good catch, id make an issue for that @kaukas_gitlab
<FromGitter> <tenebrousedge> ^
alex``` has quit [Quit: The Lounge - https://thelounge.chat]
<FromGitter> <tenebrousedge> kind of an embarrassingly simple bug :/
<FromGitter> <Blacksmoke16> :shrug:
<FromGitter> <ilanpillemer> maybe if I let my solution runs for hours it will work
<FromGitter> <Blacksmoke16> oh nice, so will be fixed next release
<FromGitter> <tenebrousedge> I'm still hoping that this will be fixed someday https://github.com/crystal-lang/crystal/issues/7577
<FromGitter> <Blacksmoke16> @tenebrousedge anyway, true; having one orm obj at a time would be ideal, have access to all the extra methods/functionality
<FromGitter> <ilanpillemer> amazing that something that looks so simple explodes so much
<FromGitter> <Blacksmoke16> i.e. in my use case having 1 full obj in memory at a time would be better than 12,000 :P
<FromGitter> <tenebrousedge> for sure
<FromGitter> <ilanpillemer> streaming can save your day
<FromGitter> <Blacksmoke16> oh?
<FromGitter> <ilanpillemer> thats why.. sed
<FromGitter> <Blacksmoke16> hm?
<FromGitter> <ilanpillemer> also map-reduce
<FromGitter> <tenebrousedge> D: sed
<FromGitter> <ilanpillemer> yes. thats why it exists
<FromGitter> <ilanpillemer> [s]treaming [ed]itor
<FromGitter> <tenebrousedge> I just got on the vim train T__T dun make me sed. awk is also sadness
<FromGitter> <Blacksmoke16> oh sorry i thought you were referring to my db issue :P
<FromGitter> <ilanpillemer> its the same old issue
<FromGitter> <ilanpillemer> as old as computer and memory
<FromGitter> <ilanpillemer> and more stuff than memory
<FromGitter> <ilanpillemer> and you need to process it
<FromGitter> <ilanpillemer> if the data base can do cursors you can use those
<FromGitter> <ilanpillemer> cursors == iterators
<FromGitter> <ilanpillemer> some dbs offer ‘offset’ and ‘limit'
<FromGitter> <ilanpillemer> thats what I am using for something db related I process
<FromGitter> <ilanpillemer> I use limit for the number of records I get at a time
<FromGitter> <Blacksmoke16> im assuming there isnt anything for that in the db driver shards :/
<FromGitter> <ilanpillemer> I then increase the offset
<FromGitter> <Blacksmoke16> but yea that sounds like it would be super helpful
<FromGitter> <ilanpillemer> until I get no more records
<FromGitter> <tenebrousedge> `offset` and `limit` will probably work, but I would hesitate to rely on that
<FromGitter> <ilanpillemer> in my case, I have no other choice
<FromGitter> <tenebrousedge> as far as I'm aware, either you're going to lock the table or you're going to have issues with writes
<FromGitter> <ilanpillemer> in general or with Crystal?
<FromGitter> <tenebrousedge> in general
<FromGitter> <ilanpillemer> Why?
<FromGitter> <tenebrousedge> because tables aren't immutable?
<FromGitter> <ilanpillemer> you mean because someone could update the table during the process?
<FromGitter> <ilanpillemer> So what I do.. is I actually stream it twice into a temporary place. And then compare and check the results are the same.
<FromGitter> <ilanpillemer> sadness
<FromGitter> <ilanpillemer> as I said.. no other choice
<FromGitter> <Blacksmoke16> doing `.all` used `52mb` doing query_each directly did `29`
<FromGitter> <ilanpillemer> my query is in gigabytes
<FromGitter> <Blacksmoke16> of memory*
<FromGitter> <Blacksmoke16> `GC.stats.total_bytes / 1_000_000`
<FromGitter> <Blacksmoke16> dunno how accurate that is/if there is a better way . but seems to be fine
<FromGitter> <tenebrousedge> @ilanpillemer what does your query do?
<FromGitter> <ilanpillemer> moves data between two systems
<FromGitter> <ilanpillemer> maybe doing some transformations
<FromGitter> <ilanpillemer> but there are no transactions I can depend on
<FromGitter> <ilanpillemer> so I cant lock the table
<FromGitter> <ilanpillemer> so I what I do.. is do it twice and make sure I get the same results
<FromGitter> <ilanpillemer> And only if I get the same results.. process and send on
<FromGitter> <ilanpillemer> otherwise there could be a write in the middle
<FromGitter> <ilanpillemer> I spent a week trying to think of a better way
<FromGitter> <ilanpillemer> I couldnt
<FromGitter> <ilanpillemer> so either my brain is too small or there isnt
<FromGitter> <ilanpillemer> distributed consistency is not easy
duane has joined #crystal-lang
<FromGitter> <tenebrousedge> can you create a temporary table?
<FromGitter> <ilanpillemer> not in the source database itself
<FromGitter> <ilanpillemer> the source database is inaccessible
dostoyevsky has quit [Quit: leaving]
<FromGitter> <ilanpillemer> except via a read query with offset and limit.. and there are no transactions
dostoyevsky has joined #crystal-lang
<FromGitter> <tenebrousedge> and you can't read the whole thing at once?
<FromGitter> <ilanpillemer> no.. as the query fails with ‘too big'
<FromGitter> <tenebrousedge> lame
<FromGitter> <ilanpillemer> yes
<FromGitter> <ilanpillemer> but I have to make do with that
<FromGitter> <tenebrousedge> so it goes :/
<FromGitter> <ilanpillemer> and then I built a complex system on top of that!
<FromGitter> <ilanpillemer> with its own mini DSL
<FromGitter> <ilanpillemer> and users can then create there own data pipes and deploy them
<FromGitter> <ilanpillemer> and boy.. did they.
<FromGitter> <tenebrousedge> I'm scared already o__o
<FromGitter> <ilanpillemer> the little shop of horrors
<FromGitter> <ilanpillemer> I was told if I give users a hammer they will hammer away
<FromGitter> <ilanpillemer> in the morning, in the evening and all over this world.
<FromGitter> <tenebrousedge> 😱
<FromGitter> <ilanpillemer> anyway, at least it has no bugs and it works.
<FromGitter> <kinxer> "it has no bugs" :O
<FromGitter> <ilanpillemer> Yup.
<FromGitter> <tenebrousedge> "it has no *known* bugs"
<FromGitter> <ilanpillemer> Well.. no known bugs after being use for quite a while 24 hours a day
<FromGitter> <ilanpillemer> I did sit and watch the logs all day for the first few weeks
<FromGitter> <kinxer> Oh, okay.
<FromGitter> <ilanpillemer> Had them scrolling by on the side
<FromGitter> <kinxer> Still impressive, but more believable.
<FromGitter> <ilanpillemer> I wonder if my simple bfs will run all night or run forever
<FromGitter> <ilanpillemer> I should try work out the O
<FromGitter> <kinxer> "simple bfs"?
<FromGitter> <ilanpillemer> oops.. I left out the “solved?” so it will run forever
<FromGitter> <ilanpillemer> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5db36ca07477946badce362a]
<FromGitter> <ilanpillemer> better fix that
<FromGitter> <tenebrousedge> `each` looks like it wants to be `each_with_object`
<FromGitter> <ilanpillemer> Yes!
<FromGitter> <ilanpillemer> though I wonder if this thing will ever finish
<FromGitter> <ilanpillemer> I may need to find ways to prune the solution space
<FromGitter> <kinxer> Oh, "breadth-first search". This is the one you're doing for an AoC puzzle?
<FromGitter> <tenebrousedge> *nods*
<FromGitter> <ilanpillemer> day 11
<FromGitter> <ilanpillemer> so far the hardest day in the set for 2016
<FromGitter> <ilanpillemer> and it looks so simple
<FromGitter> <ilanpillemer> you can see the recursive piece too with the `e.parent = n`
<FromGitter> <ilanpillemer> its probably a hanoi look alike
<FromGitter> <kinxer> If you're looking for the minimum number of steps, would that solution space be limited to only those solutions who have fewer steps than whatever your current minimum is?
<FromGitter> <ilanpillemer> bfs always finds mininum steps as the frontier opens up one level at a time
<FromGitter> <ilanpillemer> but just spotted an error in my solved so restarting
<FromGitter> <ilanpillemer> Ooh.. this looks fugly ` @@discovered = Hash(Hash(Int32, Array(String)), Bool).new { |h, k| h[k] = false }`
<FromGitter> <ilanpillemer> I wonder if it even works
<FromGitter> <ilanpillemer> can you a hash as a key in a hash?
duane has quit [Ping timeout: 265 seconds]
<FromGitter> <ilanpillemer> or do I need to work out some serialisation to use as key?
<FromGitter> <tenebrousedge> it works
<FromGitter> <ilanpillemer> even if the insertion order was different?
hightower4 has quit [Read error: Connection reset by peer]
<FromGitter> <tenebrousedge> > Two objects refer to the same hash key when their hash value (Object#hash) is identical and both objects are equal to each other (Object#==)
<FromGitter> <tenebrousedge> I suspect that the hash hashes will differ
<FromGitter> <ilanpillemer> Hmm...
<FromGitter> <tenebrousedge> interesting
<FromGitter> <ilanpillemer> maybe I should use the hash as the key
<FromGitter> <tenebrousedge> see if it works o.o
<FromGitter> <ilanpillemer> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5db3745a2a6494729c3d7438]
<FromGitter> <tenebrousedge> doesn't look terribly efficient, but I think you're right
<FromGitter> <tenebrousedge> are these big hashes?
<FromGitter> <ilanpillemer> hmm.. also means the value must always be represented the same way
<FromGitter> <ilanpillemer> flip
<FromGitter> <ilanpillemer> this day is nasty
<FromGitter> <ilanpillemer> Ok.. better fix that
<FromGitter> <tenebrousedge> arrays compare each element
<FromGitter> <ilanpillemer> if array are ordered differently are they equal?
<FromGitter> <tenebrousedge> no. Sorting is a good idea
<FromGitter> <ilanpillemer> yup
<FromGitter> <ilanpillemer> thats what I am adding in now
<FromGitter> <ilanpillemer> but leaving the hash as the key I hope is ok
<FromGitter> <tenebrousedge> seems like it
<FromGitter> <ilanpillemer> Unit64 is less fugly
<FromGitter> <ilanpillemer> so this fails
<FromGitter> <ilanpillemer> ``` if v.includes? "SG" || (!v.any? { |s| s.index("G") })```
<FromGitter> <ilanpillemer> but this works
<FromGitter> <ilanpillemer> ``` if v.includes?("SG") || (!v.any? { |s| s.index("G") })```
<FromGitter> <tenebrousedge> yes
<FromGitter> <tenebrousedge> parens are your friend
<FromGitter> <tenebrousedge> also `!any?` == `none?`
<FromGitter> <tenebrousedge> `none? &.index("G")`
<FromGitter> <ilanpillemer> woohoo solved part1
<FromGitter> <ilanpillemer> finally
<FromGitter> <ilanpillemer> now I will clean up with your suggestions tomorrow (and others I can think of0
<FromGitter> <tenebrousedge> 👍
<Yxhuvud> part 1?
<FromGitter> <ilanpillemer> https://adventofcode.com/2016/day/11
<FromGitter> <ilanpillemer> day 11 part 1 aoc
<FromGitter> <ilanpillemer> 2016
<FromGitter> <ilanpillemer> I am doing the 50 coding puzzles in crystal
<Yxhuvud> ah. my solution is at https://github.com/yxhuvud/advent_of_code_2016/blob/master/linuss-crystal/src/aoc11.cr , but probably doesn't compile anymore.
<Yxhuvud> holy crap, I spent a lot of lines there :/
<FromGitter> <ilanpillemer> I glanced at asterites.. he was doing tricks with bits
<FromGitter> <ilanpillemer> wow
<FromGitter> <ilanpillemer> More than me!
<FromGitter> <ilanpillemer> and I am stilling going to tidy
<Yxhuvud> 2016, the year where I actually cared to write maintainable code rather than just solve the problem
<FromGitter> <ilanpillemer> lol
<FromGitter> <ilanpillemer> will make more succint and less silly tomorrow
<FromGitter> <ilanpillemer> and then do part 2
<FromGitter> <ilanpillemer> those if elses are ridiculous
<Yxhuvud> your not_nil!'s are ridiculous :P
<FromGitter> <ilanpillemer> yes those too!
<FromGitter> <ilanpillemer> how do I get rid of those?
<Yxhuvud> by not having nilable types. It takes practice to know how to format the code to not need them. I used a total of 6 not_nil! in 2018.
<Yxhuvud> *aoc2018
<FromGitter> <ilanpillemer> day 15
<FromGitter> <ilanpillemer> forever in my memory
<Yxhuvud> 2018?
<FromGitter> <ilanpillemer> yes
<FromGitter> <ilanpillemer> that elf goblin battle
<FromGitter> <ilanpillemer> and the particle detector part 2 (I think day 22)
<FromGitter> <ilanpillemer> those were the two I struggled with in 2018
<Yxhuvud> I'm reasonably happy with https://github.com/yxhuvud/aoc18/blob/master/day15.cr :P
<FromGitter> <ilanpillemer> look at this monstrosity
<FromGitter> <ilanpillemer> yours looks nice
<FromGitter> <ilanpillemer> (relatively)
<FromGitter> <ilanpillemer> day 23 (not 22) I had to go to reddit in the end for help
<FromGitter> <ilanpillemer> Day 23 part 2 was the only one I could not solve without hints
<FromGitter> <ilanpillemer> 2018
<FromGitter> <ilanpillemer> But I got very close.. using a stochastic random walk
<FromGitter> <ilanpillemer> I should have realised that the random walk should have snapped to a valid radius.. then I think my original algorithm might have worked.
<Yxhuvud> Yeah I also had major issues with that one.
<FromGitter> <ilanpillemer> topaz posted that day 23 part 2 he considered the most difficult puzzle of 2018
<FromGitter> <ilanpillemer> so that made me feel a bit better about it
<Yxhuvud> Yes, though getting all the details right in the elf grand melee was a nice runner up
<FromGitter> <ilanpillemer> that was just very painful
<FromGitter> <ilanpillemer> mhy final bug was a resurrecting elf
<FromGitter> <ilanpillemer> if someone moved into a dead spot, the original elf in that spot resurected
<Yxhuvud> funniest was day9. my 14 lines of code happily accepted the bigger problem while still being ridiculously fast.
<FromGitter> <ilanpillemer> I actually found the magic missile thingie from 2015 harder because I stupid and didnt do it immutable
<Yxhuvud> didn't do 1015.
<FromGitter> <ilanpillemer> yeah. day 9 I had to change to use a ring list
<FromGitter> <ilanpillemer> I think
<FromGitter> <ilanpillemer> I did it first time with an array and that failed badly
<FromGitter> <ilanpillemer> you needed a linked list I think?
<FromGitter> <ilanpillemer> cant remember properly now
<Yxhuvud> No.
<Yxhuvud> You definitely don't need a linked list :D
<FromGitter> <ilanpillemer> maybe that was a different one then
<FromGitter> <ilanpillemer> no it was that one
<FromGitter> <ilanpillemer> my array one didnt work and when i switched to linked list ring it worked
<FromGitter> <ilanpillemer> a deque in crystal is a ring linked list I think
<FromGitter> <ilanpillemer> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5db387b0f26ea4729df78cf3]
<FromGitter> <ilanpillemer> Yup.. a circular buffer
<FromGitter> <ilanpillemer> hmm
<FromGitter> <ilanpillemer> what is a dynamic array?
<Yxhuvud> Just a regular array. A deque plays with the indices so that the start and end of it isn't stationary.
<FromGitter> <ilanpillemer> when I used arrays it killed my poor machine
<Yxhuvud> If it had removed stuff in the middle it wouldn't have worked with a deque, but as long as it only deletes at either endsit is really fast.
<Yxhuvud> (or well, strictly speaking a dynamic array is one that resizes itself if it is full)
<Yxhuvud> but unless you use C, that is basically a normal array.
<FromGitter> <ilanpillemer> this is the version that was too slow using arrays
<FromGitter> <Blacksmoke16> in regards to the db iterator thing, apparently Granite already has something that does this
<FromGitter> <tenebrousedge> good to hear
<FromGitter> <tenebrousedge> tbh I was more surprised that it didn't
<FromGitter> <ilanpillemer> I didnt know an array could resize
<FromGitter> <ilanpillemer> I thought you have to create a new array and move pointers
<FromGitter> <tenebrousedge> I mean, we all get a little larger from time to time, right?
<FromGitter> <ilanpillemer> if its allocated not enough memory
<FromGitter> <ilanpillemer> then you have to create a new one with bigger capacity
<FromGitter> <ilanpillemer> and move all the stuff
<FromGitter> <ilanpillemer> and make it look like it got bigger
<FromGitter> <ilanpillemer> and that can really slow things down
<FromGitter> <ilanpillemer> but I think thats all hidden away these days in most languages
<FromGitter> <ilanpillemer> how can I reformat my code so I dont need not_nil!
<FromGitter> <tenebrousedge> use `try` instead
<FromGitter> <ilanpillemer> anyway tomorrows conundrum
<FromGitter> <ilanpillemer> ` n.floors[with_elevator.not_nil! - 1] += c.to_a`
<FromGitter> <ilanpillemer> you can use try here?
<FromGitter> <ilanpillemer> anyways
<FromGitter> <tenebrousedge> theoretically, but can you make `with_elevator` not return a nilable?
<FromGitter> <ilanpillemer> gnight
<FromGitter> <ilanpillemer> Yes. I tried and got confused.
<FromGitter> <tenebrousedge> well, sounds like a tomorrow problem
<FromGitter> <ilanpillemer> ``` def with_elevator ⏎ floors.find { |k, v| v.includes? "E" }.try &.first ⏎ end``` [https://gitter.im/crystal-lang/crystal?at=5db38a67a03ae1584f0d88b7]
<FromGitter> <ilanpillemer> how can you make that return something not nillable?
<FromGitter> <tenebrousedge> `|| 0`, at a guess
<FromGitter> <ilanpillemer> 0 is wrong
<FromGitter> <ilanpillemer> ahh
<Yxhuvud> can't you just keep the floor with the elevator in a variable and not have to search for it at all?
<FromGitter> <ilanpillemer> but it will never actually return 0
<FromGitter> <ilanpillemer> so that would work with any number actually
<FromGitter> <tenebrousedge> if it will never be `nil`, then you should do `not_nil!` there
<FromGitter> <tenebrousedge> instead of `try`
<FromGitter> <ilanpillemer> ahh
<FromGitter> <ilanpillemer> cool
<FromGitter> <ilanpillemer> things to fix tomorrow
<FromGitter> <ilanpillemer> gnight
<FromGitter> <tenebrousedge> bonus nachos