<FromGitter>
<AlexCouch> So just out of curiosity, I've tried looking around for like a roadmap for this and couldn't find anything but what's the status on the port to windows. I'd like to see crystal available for windows some time soon.
<FromGitter>
<rishavs> We all do @AlexCouch . but we are not there yet. there has been a lot of fantastic work done over the last few weeks and you can run Crystal in windows now. ⏎ Its cumbersome and not all parts of Crystal are yet ready for Windows, but you can run crystal.exe if you want to
<FromGitter>
<confact> Is `crystal docs` out of date? I get errors in the code when I run crystal docs but not when I do normal run. I get this error in crystal docs: ⏎ ⏎ ``` code paste, see link``` ⏎ ⏎ But tests and code running that code don't fail. [https://gitter.im/crystal-lang/crystal?at=5ea54ab03c03d320626b340e]
_whitelogger has joined #crystal-lang
<FromGitter>
<kingsleyh> how do you use non_blocking_select on Channel? any examples anywhere?
<FromGitter>
<xmonader> ooh! a sec will try again my bad
<jhass>
well or missing it one, ymmv
<FromGitter>
<xmonader> builds :) my bad thank you so much
<jhass>
yw :)
<raz>
travis is so slow and annoying. perhaps crystal could (optionally?) place a .circleci/ or .github/actions when creating shards instead?
<jhass>
maybe we should just collect and document common CI configurations somewhere and not put any into init. Back then it made sense because it was basically the only well supported option
<raz>
yup, super minor thing anywa
<raz>
y
<raz>
i think writing out a default is actually great (saves everyone from having to find/cobble it together themselves)
<raz>
i'm just surprised why it's travis, as that's (imho) the slowest of the bunch
<FromGitter>
<asterite> Kingsley Hendricks I think nonblocking select is when you have an else in the select, not sure
<jhass>
travis going downhill is kind of a tradegy but was kinda to expect after they got bought by Idera. Many of their original staff left as a result
<raz>
yup, circle seems to be the least bad option atm. i'm trying to get warm with github actions but their UI sucks (even more than circles) and crystal prob shouldn't tie itself to github.
<raz>
there may be other better ones that i'm not aware of (ci tools seems to be dime a dozen nowadays)
<raz>
one feature that i really love about circle (not sure if others have it) is the ability to run locally. 'circleci local execute' is just so much quicker than endless force-pushes.
<oprypin>
raz, github actions is by far the fastest
<oprypin>
and circleci is free basically only for linux
<raz>
yup, i noticed that too
<oprypin>
i like github actions ui. no frills.
<raz>
but no local runs (i think) and their ui...sigh
<oprypin>
whats so bad
<raz>
well, yes it's clean, but too many clicks. it needs a "next build" button
<oprypin>
"circleci local execute" wow neat
<Stephie>
i think i'd like to consolidate CI to github actions too
<Stephie>
github allows you to host your own executors...
<raz>
oprypin: careful. when you've tried it once, you don't wanna live without it anymore ;0
<raz>
their yaml syntax is awful... but those local runs are so damn helpful
<oprypin>
i really really fear the current CI setup. the scripts are huge
<oprypin>
i mean in crystal repo
<raz>
i never looked at that. in my experience it's usually best to wrap it all up (as far as possible) under a 'make ci' task. and call only that from the ci itself
<oprypin>
raz, btw what do you mean by "next build"?
<oprypin>
raz, one script doesnt look nice or allow for granularity
<raz>
oprypin: you watch a build. it fails. you re-push a fix. and want to see the new build. but there's no button. it takes 3 clicks to get to the new build :(
<raz>
oprypin: well yea, really depends on the details. but generally i prefer to keep the logic in a Makefile/script than tying it to a CI (which also makes it hard to switch CI later)
<oprypin>
i know i know
<oprypin>
it's kinda arguable though. i'd rather port a raw travis file than have to figure out what the CI script does
<raz>
yup yup, it depends...
<jhass>
I don't know, I'm not a too big github actions fan either. Their documentation really sucks. The "marketplace" is swamped with very mediocre, very specific usecase stuff
<jhass>
and their flexibility breaks down as soon as you step outside the preplanned path
<jhass>
I found no better way to skip a nightly build if there's no new commits other than having it trigger through another workflow doing an API call to do so. I mean, common
<jhass>
oprypin: well the action publishes something in an fdroid repo, so I care updating to the samme build everyday :P
<oprypin>
u can skip the deploy step
<oprypin>
i dont even know how you'd figure out "nothing changed"
<oprypin>
with a named cache or something?
<jhass>
it also continously updates a tag so I just check if the current commit already has that tag
<jhass>
well there's this macro language you can echo to stdout to set an env var and then skip steps based on that I guess. But that you can use that outside custom steps, so inside a normal run step is completely undocumented and I only discovered that by accident since I did the current setup
<jhass>
idk, the whole thing doesn't seem very thought out to me
<oprypin>
jhass, just dont use shell: cmd for that step and you're fine
<oprypin>
u can use `shell: bash` and that's totally cross platform
<oprypin>
or dont specify a shell, the default is powershell and it often happens to match bash syntax
<jhass>
I'm just saying there' so many pitfalls and no coherent documentation of it all
<oprypin>
yea but free VMs
<jhass>
like the other day my build started to randomly run out of disk space. Multiple forum threads on this for weeks now with no official reply. I rearrenged the workflow into two jobs and now it's working again. Why? No clue
<raz>
i think we can cut this short: github actions is not ready yet. :)
<raz>
my vote would go to circle (from the choices that i'm aware of). it has plenty warts, too, but they've been around long enough so that most of the stuff that people care about is solid.
<FromGitter>
<stellarpower> Hey, does anyone know if there are any good shards that can be used for network (diagnostic) stuff? Things like pinging, figuring out the current machine's address, gateways, route tracing, DNS lookups?
<jhass>
I still think we just shouldn't generate one. Travis is there because history, it used to be the first (to exist in the style of modern integrated CI/CD platformss and to have (community provided) support for Crystal). Don't get me wrong, I do use Github Actions. I think currently there's just no obvious choice for Crystal, it depends on your needs. So we should just document the options
<jhass>
somewhere central and give some copy & paste starting points.
<FromGitter>
<stellarpower> I usually start scripts like this in the shell, and then they get more complicated, and then I regret not investigating more first into a more programmatic way of doing it.
<raz>
jhass: i disagree with that. i think it's awfully convenient to generate one by default and i think it helps CI adoption (even travis is better than nothing).
<FromGitter>
<kingsleyh> hello - does anyone have any tips on the best way to spawn multiple workers that report back information that is collated at the parent? I was using Process.fork which was working fine and started each worker in a new os process - but since it's going away soon - I was trying to use the preview_mt stuff - but so far not having much luck - I'm getting a lot of Channel closed errors and sometimes the channel
<FromGitter>
... just stops receiving anything.
<FromGitter>
<kingsleyh> also it doesn't seem to matter how many workers I spawn - the activity monitor in MacOS says there are 23 threads
<FromGitter>
<kingsleyh> I had thought that value would change at some point
<FromGitter>
<stellarpower> The whole design of shards is that it's distributed right? There's no central repo, it will search git or other sources to find what you ask for?
<jhass>
pretty much. And shards itself will not search, you just give it the full URL to clone from
<jhass>
it'll probably support other mechanisms than git too in the future
<jhass>
I kinda like the idea to have a defacto central registry for discovery, ratings & usage stats though. There's just no reason to make the tool depend on it
<jhass>
and as long as shards install still works with it down, I wouldn't even mind a shards search, shards add etc that makes use of it, personally
<jhass>
discovery is the arch nemesis of distributed designs :/
<oprypin>
kingsleyh, u can try Process.new and UNIXSocket
<FromGitter>
<stellarpower> That's what I was gonna say, some of these duplicated sites look sort of official but if they aren't may be better to emphasise it.
<raz>
i always use crystalshards.xyz
<raz>
the copycats should just shutdown, they only confuse new users
<FromGitter>
<asterite> none is official
HumanG33k has joined #crystal-lang
HumanG33k has quit [Remote host closed the connection]
HumanG33k has joined #crystal-lang
zorp has quit [Ping timeout: 260 seconds]
<FromGitter>
<xmonader> `/a(?<grp>sd)f/.match("_asdf_").try &.["grp"]` i want to capture the match in an object and access the subgroups ⏎ ⏎ ``` code paste, see link``` ⏎ ⏎ this doesn't seem right [https://gitter.im/crystal-lang/crystal?at=5ea5c31661a0002f794e3ba2]
<FromGitter>
<kingsleyh> oprypin - thanks I will look into it
<FromGitter>
<lbarasti> > I'm getting a lot of Channel closed errors and sometimes the channel just stops receiving anything. ⏎ @kingsleyh the channel closed errors might be an indication that you're closing a channel before the consumer has read whatever it was supposed to read from it. ⏎ Another possibility is that you actually want the channel to close, but you're not handling the exception on the consumer side. You can
<FromGitter>
... address that with a simple `rescue`. ⏎ Regarding "the channel just stops receiving anything", that's sometimes a symptom of the fact that the consumer is not consuming, hence the channel is full. As a consequence, the producer gets stuck, and the program stalls. One simple thing to check: is the consumer running in its own Fiber? [https://gitter.im/crystal-lang/crystal?at=5ea5c50d3ea9b172a48fca46]
<FromGitter>
<lbarasti> > also it doesn't seem to matter how many workers I spawn - the activity monitor in MacOS says there are 23 threads ⏎ ⏎ in my understanding, even when you enable multi-threading, the number of threads allocated to your crystal program is fixed - see https://crystal-lang.org/2019/09/06/parallelism-in-crystal.html ⏎ So it's totally fine that the number if threads will not go up. That's the beauty of it,
<FromGitter>
<Blacksmoke16> it defaults to 4 but can be overridden via an env var
<FromGitter>
<lbarasti> 👍
<FromGitter>
<jojokoro> Is right assume `def self.method`in a Crystal class is equivalent to "static" methods in other languages?
<FromGitter>
<Blacksmoke16> eys
<FromGitter>
<Blacksmoke16> yes
<FromGitter>
<kingsleyh> @lbarasti thanks for the advice - do you think it's better to try to use the multi threading - or try to stay with starting up new processes - without using Process.fork?
<FromGitter>
<xmonader> if i've an object O1 who has name property that creating another O2 object that has a `name` property ⏎ ⏎ I should be able to do so in the body of O1 ⏎ ⏎ ```O2.new(@name = @name) ``` right? [https://gitter.im/crystal-lang/crystal?at=5ea5d1122bf9ef1269a68f24]
<FromGitter>
<lbarasti> @kingsleyh I'd start with no multi-threading and no new processes, then try and enable multi-threading with the default number of workers
<FromGitter>
<kingsleyh> @lbarasti I need a way to let the user ramp up or down their cpu utilisation
<FromGitter>
<kingsleyh> It’s a blockchain miner
<FromGitter>
<kingsleyh> And currently it uses Process.fork
<FromGitter>
<Blacksmoke16> that internally spawns the same program multiple times?
<FromGitter>
<Blacksmoke16> couldnt you just leave that up to them
<FromGitter>
<lbarasti> could you run multiple instances of the app, rather than increasing the number of processes running in each?
<FromGitter>
<PercussiveElbow> Anyone have a recommended background job library? Sidekiq seems kind of broken currently and mosquito is missing the ability to use multiple queues.
<FromGitter>
<kingsleyh> @PercussiveElbow I've not used it but I saw something called Dispatch earlier
<FromGitter>
<Blacksmoke16> prob could build some abstraction on top of it
<FromGitter>
<kingsleyh> @lbarasti It's possible that they could just run multiple instances of the app - but it might mean running a huge amount of them - and then it becomes a bit cumbersome to start and stop them all etc
<FromGitter>
<Blacksmoke16> MT would prob be way to go, then it could be started like `CRYSTAL_WORKERS=10 ./bin/app`
<FromGitter>
<Blacksmoke16> however its still experimental, and unless you were using fibers internally already prob would require a good amount of work id imagine
<FromGitter>
<kingsleyh> what is a CRYSTAL_WORKERS exactly? I didn't quite understand from the documentation
<FromGitter>
<Blacksmoke16> the number of worker threads
<FromGitter>
<kingsleyh> but what is a worker thread? some internal crystal thread?
<FromGitter>
<Blacksmoke16> something that does work
<FromGitter>
<Blacksmoke16> sec
<FromGitter>
<kingsleyh> I initially thought it was how many cores, but it's not, and then I thought it was how many os threads - but it's also not!
<FromGitter>
<Blacksmoke16> it is tho, if your cpu has 4 cores and 8 threads, then you would set `CRYSTAL_WORKERS` to `8` in order to use 100% of the cpu
<FromGitter>
<kingsleyh> is it usually 1 thread = 1 core?
<FromGitter>
<Blacksmoke16> depends on the cpu
<FromGitter>
<kingsleyh> how can I tell how many threads my mac has? it has 10 cores
<FromGitter>
<Blacksmoke16> what cpu does it have?
<FromGitter>
<kingsleyh> what happens if I set it to 500 threads - it will just cap at 20 and not crash?
<FromGitter>
<lbarasti> @kingsleyh the number of CRYSTAL_WORKERS is the number of Operating System threads that will be allocated to your application. It is OK to exceed the number of physical cores of your machines, but you might see a degradation of performance if you set the number too high
<FromGitter>
<Blacksmoke16> dunno
<FromGitter>
<Blacksmoke16> prob would just tank
<FromGitter>
<kingsleyh> @lbarasti ah ok thanks - I set it to 10000 and it seems to be running
<FromGitter>
<Blacksmoke16> lol
<FromGitter>
<Blacksmoke16> thats probably not wise
<FromGitter>
<kingsleyh> although setting it to 100 seems to get the CPU far higher
<FromGitter>
<kingsleyh> but I can't seem to get the CPU over 400%
<FromGitter>
<lbarasti> 10000 is the kind of number where the contention for resources will slow the app down
<FromGitter>
<Blacksmoke16> default is 4, so sure you're setting it correctly?
<FromGitter>
<lbarasti> as a rule of thumb, you don't want to exceed the number of logical cores in your system, but it's ok to try various config
<FromGitter>
<kingsleyh> I set the CRYSTAL_WORKERS to 20 - and spawn 20000 workers
<FromGitter>
<Blacksmoke16> ah ok
<FromGitter>
<lbarasti> > I set the CRYSTAL_WORKERS to 20 - and spawn 20000 workers ⏎ that should be all right
<FromGitter>
<lbarasti> 20 might give you worse performance than 8, though. Worth trying both
<FromGitter>
<kingsleyh> I think if I can fix the issues with multi-threading and channels - then probably preview_mt would be easier than starting multiple Process.new
<FromGitter>
<kingsleyh> maybe 1 Channel is not sufficient?
<FromGitter>
<Blacksmoke16> you'll now need to make sure your code is thread safe tho
<FromGitter>
<kingsleyh> hmm yeah probably take a while to get that working
<FromGitter>
<kingsleyh> I'm not very knowledgeable about multithreading
<FromGitter>
<kingsleyh> I'll do some experimentation anyway - I could keep using Process.fork in the short term till it goes away
<FromGitter>
<Blacksmoke16> 👍
<FromGitter>
<PercussiveElbow> > @PercussiveElbow I've not used it but I saw something called Dispatch earlier ⏎ ⏎ Thanks I'll check that out, though I think it might not work for my use case as the workers would be on different hardware.
<FromGitter>
<Blacksmoke16> mosquito could work no? each job has its own queue in redis
<FromGitter>
<kingsleyh> @lbarasti is there a way to log what thread something is running in?
<FromGitter>
<Blacksmoke16> try like `Fiber.current.name`
<FromGitter>
<Blacksmoke16> prob will be `nil` tho
<FromGitter>
<kingsleyh> @lbarasti what does select do on a Channel?
<FromGitter>
<kingsleyh> and what is non_blocking_select
<FromGitter>
<kingsleyh> is it like a peek?
<FromGitter>
<kingsleyh> @Blacksmoke16 ^^
DTZUZU has quit [Quit: WeeChat 2.7]
<FromGitter>
<kingsleyh> @lbarasti - also what does capacity mean on Channel.new( ??? )
<FromGitter>
<Blacksmoke16> select allows waiting for a value from multiple channels
<FromGitter>
<lbarasti> > is there a way to log what thread something is running in? ⏎ ⏎ Yes, as @Blacksmoke16 says. To avoid it being nil, you can pass a name to the fiber when spawning: `spawn(name: "my-fiber") do #...`
<FromGitter>
<lbarasti> alternatively, you can use `Fiber.current.object_id`, not very human friendly, though
<FromGitter>
<lbarasti> > what does select do on a Channel? ⏎ ⏎ It allows you, among other things, to run different blocks depending on which channel receives or sends first. ⏎ It also allows to provide a default operation to be run whenever all the remaining clause are "blocked". ⏎ It's a slightly advanced construct, I suggest you familiarise yourself with the basics first. [https://g
<FromGitter>
<lbarasti> > also what does capacity mean on Channel.new( ??? ) ⏎ the best way I can explain this is in this "intro to concurrency" I recorded recently: https://www.youtube.com/watch?v=hntKpUKNtLw
<FromGitter>
<kingsleyh> @lbarasti is there a way to kill a Fiber? or make it stop somehow?
<FromGitter>
<kingsleyh> what does Fibre.inactive do?
ur5us has joined #crystal-lang
_ht has quit [Quit: _ht]
<FromGitter>
<ImAHopelessDev_gitlab> crystal time baby
<FromGitter>
<ImAHopelessDev_gitlab> @lbarasti nice video! very well-spoken, i'm 5 mins in, but will watch rest l8er
<raz>
hm. i've been using crystal for 10 years or so but never once found a need for channels
<raz>
the day will come, i'm sure
DTZUZU has joined #crystal-lang
<FromGitter>
<lbarasti> thanks @ImAHopelessDev_gitlab! @kingsleyh the fiber will die once the block you pass to `spawn` is over. You often `loop` inside that block, though, e.g. to read value from a channel. ⏎ Hence a `Channel::ClosedError` exception being raised in the loop *can* be used as a termination condition
<FromGitter>
<lbarasti> calling it a day, good night!