<unshadow>
About the file autoclose issue, I think that if you use File.read or File.write thats ok, but, File.open returns the file handler, where even in Ruby its custom do a "f = File.open..." and then f.read\write and in the end f.close
<unshadow>
Or if you use it in a block then it's also make sense to autoclose it after the block
<fowlduck>
state machines, lexing, and parsing oh my!
<fowlduck>
jesus
<fowlduck>
I mean, I'm sure this all makes sense
<fowlduck>
translate RFC => code
<fowlduck>
have you tried it with multiple telnet servers' output yet?
<unshadow>
Right now I'm stuck becuase of an apperent Compiler bug
<fowlduck>
I had a colleague who wrote a telnet client in python from scratch who had trouble with the nonsense differences between servers
<fowlduck>
oh
<fowlduck>
well damn
<fowlduck>
it's god's work you're doing, though :)
<fowlduck>
hey, has anyone mentioned work on an actor lib?
<unshadow>
nha.... it's super selfish , I need it so I do it, I dont think anyone really need a Telnet lib
<fowlduck>
hey, I'm sure some hipster hacker with a botnet somewhere might be able to use it ;)
<fowlduck>
don't they use that for command & control?
<unshadow>
Building a raw tcp protocol is much easier...
<unshadow>
just socket.read, parse, execute, send back reply
<fowlduck>
I've never done that, either, but that's surprising
<fowlduck>
huh
<fowlduck>
I guess that makes sense
<fowlduck>
that ought to be worked out by now
<fowlduck>
or wait, socket.read, you're not reading bytes off the line, so you mean a new protocol
<fowlduck>
I misunderstood
Ven has joined #crystal-lang
<unshadow>
I mean just creating a parser for cleartext data over the TCP connection ;)
<jhass>
I thought C&C is all IRC if you're too lazy to do your own?
<fowlduck>
oh, right, IRC, not telnet
<unshadow>
data = socket.read, case data while /kill all humanity/ ; exec etc...
<fowlduck>
oh, that reminds me, I need to make that github project private
<jhass>
heh
<unshadow>
C&C on IRC is usually botnet for DDOS and "quick'n'dirty" attacks, while, "off the grid" botnets for APT attacks and special operations are usually ran thourgh TOR with the use of customized protocol
<unshadow>
We found a very massive one using one of our customers servers, which apperently was hacked and enlisted to the botnet
<unshadow>
jhass: why dons't the bot announce PR's ? usually it did right ? also stoped telling about build success failure
<jhass>
oh, I guess the used notice mode
<jhass>
meh
<jhass>
we'll see when it becomes a problem I guess
datanoise has joined #crystal-lang
<unshadow>
fowlduck: I got interested in crystal becuase of a project in work, we usually use Ruby, but, above 10K concurrent connections the issues with GIL\GVL was drasticlly killing the performace as we couldent use more then 1 CPU Core at a time, Now I add slowly everything I need to the project to Crystal in the hopes of starting a full code migration in a couple of months
havenwood has quit [Ping timeout: 252 seconds]
<fowlduck>
using telnet?
<fowlduck>
10k telnet connections? tell me it's true
<fowlduck>
that sounds about right for ruby, though
<unshadow>
botnet much ? hahahha
datanoise has quit [Ping timeout: 245 seconds]
<fowlduck>
hahaha
<fowlduck>
we've done a stupid amount of work to setup pipelines between ruby and go to handle massive numbers of connections
<fowlduck>
over redis, even
<fowlduck>
seeing the benchmarks for crystal, I'm interested to see how it handles this
<unshadow>
kidding though, our software is a reverse proxy which scans incoming data for application and network level attacks, it supports multi protocols like HTTP/HTTPS, SSH, SFTP, SMTP, RDP each has it's own attack surface and techniques and we perform deep packet inspection etc.. to find malicious data
<fowlduck>
oof, 33 pull requests are open
<fowlduck>
whoa, cool
<fowlduck>
I don't do anything near that level
<unshadow>
fowlduck: what do you guys do ?
<fowlduck>
mostly using ruby libraries and binary tools to handle tons of protocols and services
<fowlduck>
and all the minutia around dealing with fan-out shared-resource-hungry processes
<unshadow>
fowlduck: Are you guys useing Puppet ?
<fowlduck>
I see safe-t and I immediately don't trust it hahaa
<unshadow>
XD
<fowlduck>
we use chef but we have to decide when to scale up and when to scale down transcoding resources
<fowlduck>
and predict resource usage for given transcodes so we can efficiently pack them onto our transcoding VMs
<fowlduck>
and pack these processes fairly amongst accounts
<fowlduck>
so no one account can starve the others
<fowlduck>
there's a tremendous amount of orchestration
<unshadow>
fowlduck: thats sounds really cool ! so it's like a live action IT\devops job :)
<fowlduck>
ha, yeah, I do plenty of ops-y stuff
<fowlduck>
one of our guys wrote a machine learning algorithm to handle our transcoding worker autoscaling
<fowlduck>
or copied one that existed, rather
<fowlduck>
it seems to have worked out well
<fowlduck>
plus we deal with making our resource usage efficient, so making sure we use spot EC2 workers, making sure we kill workers at the right time, etc
alsm has left #crystal-lang ["Leaving"]
<fowlduck>
and a lot of the behavior of the system is emergent. there is no central brain for the whole thing, so most of the things we do are complex systems that are gently manipulated to produce different emergent behavior
sandelius has joined #crystal-lang
<fowlduck>
it's pretty wild when I really think about it
<unshadow>
XD it's great to have a job you like and enjoy
<fowlduck>
yeah, most definitely
<unshadow>
jhass: your'e a student right ?
<jhass>
yeah
<fowlduck>
unshadow: you're using crystal to write this project?
<BlaXpirit>
so much talking, i don't think i can read it all
<jhass>
happens
BlaXpirit_ has joined #crystal-lang
BlaXpirit has quit [Killed (kornbluth.freenode.net (Nickname regained by services))]
<unshadow>
haha yhe a
<asbradbury>
some of the stdlib code uses `getter :foo` rather than `getter foo` (which the docs seem to prefer). Is one considered preferable to the other?
<jhass>
these days the latter is preferably, though there's really no difference in the generated code, it's style
<unshadow>
I like :foo beter as it looks more Ruby
absurdity1 has joined #crystal-lang
<asbradbury>
I'm surprised they don't seem to be implemented as a macro, unless I'm missing it
<asterite>
jhass: about breaking changes regardless of the version, we are strictly following semver ;-) http://semver.org/
<asterite>
"Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable." :-D
<asterite>
I'm just lazy to increase a version number because of a breaking change (like Signal)... seems like "Wow, they incremented the number! ... Oh, because of a small breaking change"
<jhass>
at diaspora we (well I) tricked and hacked semver by prepending a version
<jhass>
so we got r.x.y.z
<jhass>
and do strict semver for the x.y.z part
<jhass>
but bump r on our consideration
mdz_ has joined #crystal-lang
<asbradbury>
asterite: are you working full time on crystal?
<asterite>
Mmm... not sure I like it
<jhass>
eh, not saying it would fit for crystal
<asterite>
asbradbury: no, not at all, we are doing it in our spare time
<asterite>
(well, not all of our spare time, of couse :-))
<asterite>
jhass: any idea why Dir["./**/*.cr"] doesn't behave like in Ruby?
<jhass>
yeah looks like everyone does it on their own :/
<asterite>
Don't worry, it's not a big deal. Someday we'll fix it
<asterite>
asbradbury: why you ask?
<asbradbury>
asterite: just curiosity
<jhass>
asterite: easiest would probably to "just" recursively walk everything and filter the result with fnmatch(3)
<asterite>
seems ruby calls fnmatch too
<asterite>
but looking at dir.c it looks like a huge code to port, or to do something similar... at least for now :)
<jhass>
hrm, though you'd want to know if your current directory matches part of the pattern
<jhass>
so you can decide to walk it or not
<unshadow>
asterite: What do you think about the wierd Regex thingy ?
<jhass>
so you also want to tokenize the pattern on / I guess
<crystal-gh>
[crystal] asterite pushed 1 new commit to master: http://git.io/vI64h
<crystal-gh>
crystal/master 5c10d5f Ary Borenszweig: Put back the old Regex constructor, and fix the literal expander for good so next version we can remove it. Fixes #781
<asterite>
jhass: I didn't think yet how would one implement it, probably tokenize the pattern, yes
<fowlduck>
from `ready_channel = channels.find &.ready?` to `ready_channel = channels.select(&.ready?).sample`
<fowlduck>
so it randomly selects amongst the ready channels, like go does
<fowlduck>
I don't understand how this works with concurrency, though. it seems like it shouldn't
<fowlduck>
I guess the Scheduler thing handles it?
<fowlduck>
only, what's the scheduler?
<waj>
is that actually so important? I mean… the `sample` call will add a small but noticeable overhead
<fowlduck>
is concurrency cooperative in crystal?
<fowlduck>
it really is important
<fowlduck>
otherwise the first channel will always be selected if it's always ready and that will starve out other channels that might be ready
<fowlduck>
at least that's the design motivation in go
<fowlduck>
and I think it's a solid decision
<waj>
well… yeah, in Erlang the order of the select clauses defines the priority
<fowlduck>
right. with go you would nest selects to enforce priority
<waj>
I mean… in the “receive” statement
<fowlduck>
right
<waj>
I see
<fowlduck>
so in go you at least have the *choice* of not starving out other channels
<fowlduck>
otherwise you'd have to randomize the order of them on every loop
<waj>
however, I’d like to find a way that doesn’t involve array allocations
<fowlduck>
I'm all ears
bhishma has quit [Ping timeout: 256 seconds]
<fowlduck>
I'm not sure how you could get away with not allocating an array or something else, there
<fowlduck>
if we could randomize the order of the array, or at least randomize the order we iterate over it?
<waj>
still… if your code is running on a single fiber and there is no physical time to attend all the requests on all the channels, there is always going to be messages accumulating on many channels
<waj>
I know that randomizing the selection might be useful in some scenarios, but I’m not sure it isn’t possible to live without it
<fowlduck>
the person doing the select could always randomize the order of channels themselves if it's important, I guess
<fowlduck>
do you happen to know how Scheduler works?
<waj>
this is something I’ve been thinking for a while when I implemented the Channel.select, but I wasn’t sure, so I’m all ears :)
<crystal-gh>
[crystal] asterite pushed 3 new commits to master: http://git.io/vI1kS
<crystal-gh>
crystal/master 08beee0 Ary Borenszweig: Fixed #783: special vars didn't work when blocks were involved
<crystal-gh>
crystal/master a9cb445 Ary Borenszweig: Fixed #784: bug when creating macro with argument with default value and splat
<crystal-gh>
crystal/master 14e9cb5 Ary Borenszweig: Small bool initialization in json lexer
<jhass>
ah looks like I do
<fowlduck>
jesus, controlling the scheduler really makes channels *much* easier to implement
<fowlduck>
yay single threading
<waj>
more threads will come, eventually
<waj>
however, if so many people is happy with nodejs, why not be happy with Crystal that gives you the power of evented IO without all the trouble of having callbacks everywhere :)
<waj>
fowlduck: I found the channels implementation to be quite similar (in structure) to Go channels
<waj>
anyway… it’s still really simple because it’s missing all the support for multiple threads
<strcmp1>
yeah, it is riding on hype, it will be around to stay but i think it has more hype than anything else. in a few years we will see. same as rails 7-8 years ago.
<waj>
also, the “select” must change a lot to support multithreading
<waj>
even the API interface, although we don’t know how to do it yet