lopex has quit [Quit: Connection closed for inactivity]
josh-k has quit [Remote host closed the connection]
<brixen>
johnmuhl: where "break" means what exactly?
frzng has quit [Quit: frzng]
<johnmuhl>
brixen:
<johnmuhl>
the puma workers just die eventually
<johnmuhl>
no log or response to the browser at all
stass has quit [Ping timeout: 272 seconds]
|jemc| has joined #rubinius
stass has joined #rubinius
meh` has quit [Ping timeout: 245 seconds]
dimday has quit [Remote host closed the connection]
havenwood has quit []
havenwood has joined #rubinius
JohnBat26 has joined #rubinius
diegoviola has quit [Quit: WeeChat 1.1]
<djellemah>
|jemc|: Rubinius::Channel may be useful, but AFAICS it gets in the way of what I'm trying to do - Queue#close. Precisely because Channel hides mutex and condvar.
<djellemah>
|jemc|: unless Rubinius::Channel implemented close already, in which case the mutex and condvar could stay hidden. In general, I think there's always a tradeoff between hiding "messy" primitives, and making unknown future functionality hard to implement.
craigp has joined #rubinius
johnmuhl has quit [Quit: Connection closed for inactivity]
havenwood has quit []
noop has joined #rubinius
craigp has quit [Ping timeout: 256 seconds]
amclain has quit [Quit: Leaving]
nirvdrum has joined #rubinius
nirvdrum has quit [Ping timeout: 272 seconds]
craigp has joined #rubinius
craigp has quit [Remote host closed the connection]
flavio has joined #rubinius
flavio has joined #rubinius
craigp has joined #rubinius
craigp has quit [Ping timeout: 255 seconds]
goyox86 has joined #rubinius
lopex has joined #rubinius
josh-k_ has joined #rubinius
nirvdrum has joined #rubinius
goyox86_ has joined #rubinius
Thijsc has joined #rubinius
nirvdrum_ has joined #rubinius
nirvdrum has quit [Ping timeout: 265 seconds]
goyox86 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
goyox86 has joined #rubinius
nirvdrum_ has quit [Ping timeout: 265 seconds]
Thijsc has quit [Ping timeout: 252 seconds]
craigp has joined #rubinius
lbianc has joined #rubinius
Thijsc has joined #rubinius
mrb_bk has quit []
mrb_bk has joined #rubinius
josh-k_ has quit [Remote host closed the connection]
lbianc has quit [Ping timeout: 272 seconds]
josh-k has joined #rubinius
goyox86 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
lbianc has joined #rubinius
hosiawak has joined #rubinius
goyox86 has joined #rubinius
goyox86 has quit [Client Quit]
goyox86 has joined #rubinius
hosiawak has quit [Remote host closed the connection]
Bwild has quit [Quit: leaving]
flavio has quit [Ping timeout: 265 seconds]
Thijsc has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Thijsc has joined #rubinius
flavio2 has joined #rubinius
[spoiler] has joined #rubinius
noop has quit [Ping timeout: 265 seconds]
craigp has quit [Ping timeout: 255 seconds]
johnmuhl has joined #rubinius
lbianc_ has joined #rubinius
lbianc has quit [Ping timeout: 244 seconds]
lbianc_ is now known as lbianc
pwh has joined #rubinius
|jemc| has quit [Ping timeout: 246 seconds]
unreal has quit [Ping timeout: 272 seconds]
pwh has quit []
<cremes>
what wonders will IO#gets teach me today?
<cremes>
3 broken specs to go on IO#gets but they look ugly…
unreal has joined #rubinius
|jemc| has joined #rubinius
pwh has joined #rubinius
josh-k has quit []
goyox86 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
goyox86 has joined #rubinius
<djellemah>
So I started looking at Rubinius::Channel. Talk about jumping in the deep end to get your toes wet...
goyox86 has quit [Client Quit]
<|jemc|>
djellemah: you've probably looked at this possibility more than I have then ;)
<|jemc|>
for me so far it has merely been a passing thought, so I hadn't researched issues like what features might be missing to implement Queue
<|jemc|>
It just seems silly to me to implement the Queue, a very valuable primitive for lockless concurrency, in terms of locks ;)
goyox86 has joined #rubinius
<|jemc|>
locks: my apologies, I'm trash-talking you again
lbianc_ has joined #rubinius
<djellemah>
|jemc|: Yeah, Channel would mostly be a good base for Queue.
<djellemah>
|jemc|: I realised a few years ago that Queue needs a close method. The poison-token is quite horrible really. But for close to work properly, it has to use the mutex and condvar inside Queue.
lbianc has quit [Ping timeout: 245 seconds]
lbianc_ is now known as lbianc
<|jemc|>
djellemah: the poison-token technique is bad because you might want to accept arbitrary tokens?
<|jemc|>
if that's the problem, you could look at Rubinius' special 'undefined' value - it's unlikely that your user Ruby code is using it, so it might make a better poison token than nil
unreal has quit [Quit: Very funny Scotty. Now beam down my clothes!]
<djellemah>
|jemc|: Have you ever tried getting a clean queue shutdown with multiple producers and multiple consumers? Ewww. Either the producer side has to know how many consumers there are (unnecessary coupling), or the consumer side has to repeatedly re-queue the token.
<|jemc|>
djellemah: I see, but perhaps if the mechanism was tucked away inside the implementation of Queue -
<|jemc|>
that is, Queue#pop blocks for a value from the channel, but if the value is 'undefined', it closes instead
<|jemc|>
and the Queue itself can know how many consumers there are?
<|jemc|>
I'd have to look at it more closely - I may be missing part of the problem
<djellemah>
|jemc|: hmm. Good point on the undefined. I hadn't thought of that.
<brixen>
johnmuhl: do you have rbx logging set up? should be /var/log/rbx.log, $TMPDIR/rbx.log, or /tmp/rbx.log
<brixen>
johnmuhl: it would be pretty odd for puma to exit unexpectedly with no trace at all
<brixen>
johnmuhl: eg I'd expect some sort of fault or some msg from puma about exiting
<djellemah>
|jemc|: Queue itself only knows how many consumers when they're all waiting. Which means the producer side needs to know about the consumer side again. Also eww IMO.
<brixen>
|jemc|: you can't use undefined outside of kernel
<brixen>
I mean, you can if you manually tweak the runtime compiler to include that transform
<brixen>
but the undefined transform isn's generally available
<brixen>
oh, I guess you could use Module#dynamic_method to access it
<johnmuhl>
brixen: I can't immediately find any of those logs. I'll try it again
<|jemc|>
brixen: yeah, I was about to mention dynamic_method - it may be ugly, but it's not out of the question :)
<johnmuhl>
every request causes "Terminating timed out worker"
<djellemah>
brixen: Clean queue shutdown. I can't think of a better data struture, but that wouldn't be the first time ;-)
<brixen>
johnmuhl: why are you using worker model?
<brixen>
johnmuhl: just curious
<johnmuhl>
I guess it was part of the puma config I copied from Heroku :)
<brixen>
oh, ok
<brixen>
well, you're using threads too
<brixen>
just just curious
<brixen>
so, question is, why does Puma think the worker is timed out
<brixen>
is this app something you could put up on github for me to look at?
<brixen>
s/just just/was just/
<johnmuhl>
hmm if I start it in single mode it works again
|jemc| has quit [Quit: WeeChat 1.0.1]
<djellemah>
brixen: from another angle: you can close pipes, files and network sockets. Go can close channels. unix doesn't really use ^Z as an end-of-stream marker. Close is a state, really, not a data item.
|jemc| has joined #rubinius
goyox86 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
meh` has joined #rubinius
<djellemah>
|jemc|: AFAIK, pretty much every so-called lockless concurrency technique uses atomic operations at some level. Except for ring buffer, which is limited to single-producer, single-consumer and sometimes loses data.
<djellemah>
|jemc|: Did you figure out the gitter-to-irc bridge? Please tell me how?
<|jemc|>
djellemah: I use weechat, and there is a gist for it that may help you
<brixen>
there's a bunch of stuff in that about queues and synchronization
amsi has joined #rubinius
<djellemah>
I was thinking atomic more along the lines of CAS, DCAS, LL/SC and and similar - machine code instructions that are guaranteed to not be interleaved.
<cremes>
down to 1 error in gets_spec but i think it’s a bug in String#index with encodings… working on a small repro for brixen
<brixen>
djellemah: just making the point that you can linearize with CAS but what you linearize may not need to be "atomic"
<brixen>
djellemah: but that book I referenced is really good
<djellemah>
|jemc|: Yeah, predictable and consistent interface and semantics is almost always a good thing.
<brixen>
cremes: sweet
havenwood has quit []
<djellemah>
brixen: Just glanced at the book. I'll have to look at it some more ;-)
<djellemah>
brixen: Let me try to linearize my thoughts around queue shutdown...
<brixen>
djellemah: indeed, as in, read every word in it :)
<djellemah>
brixen: You guys are so lucky to have same-day delivery. Takes at least a week here, and shipping usually costs more than the book.
<brixen>
djellemah: and do you know anyone at a university that may be able to borrow it and lend it to you?
<djellemah>
brixen: Don't worry, still here.
<brixen>
djellemah: I mean, where are you located? :p
<djellemah>
South Africa
<brixen>
ahh
<brixen>
yeah, I've been trying to only buy kindle books and it's frustrating when they are only paper
<djellemah>
In a small town by the sea. Which is awesomely pretty, but sometimes material things are hard to get hold of.
<djellemah>
Nearest university is about 250kms, but I do someone who might be able to access the library. Thanks for the suggestion ;-)
<djellemah>
s/do/do know/
<slaught>
djellemah: does SA have interlibrary loans? in the US the local library can borrow books from other libraries or universities and get the book for you.
<slaught>
djellemah: it iss how I got a lot of tech books when I was in a small town.
<djellemah>
OK, so queue shutdown. Longish running process that does several batch-ish jobs on potentially huge files where I'm trying to keep memory usage down. Well there's one assumption I have to check - will the waiting threads and the queue prevent one another being garbage collected?
<djellemah>
slaught: Another idea I didn't think of. Thanks. I'll try it.
<djellemah>
Just looking at the code to refresh my memory... brb
<djellemah>
brixen: OK, so the issue came up because the consumer side needs to know when a batch is finished so it can do some commits and flush files and so on. Can't use queue.empty? because that happens in normal usage. Poison-token is ugly (especially for multi-consumer/multi-producer) and implies unnecessary coupling. Any communication outside of the queue runs the risk of race conditions or deadlock.
<brixen>
indeed
<brixen>
could you do those "finish" operations every N items consumed?
<brixen>
or is it actually synchronized to other activities
<brixen>
and if it is, you may want to redesign that
<djellemah>
brixen: Yes, but the last batch is always odd-sized. Which means additional bookeeping.
<djellemah>
brixen: now, every time I look at producer/consumer code, the lack of a close state seems like a real drawback.
<brixen>
hmm
<brixen>
sounds like there's state shared between producers and consumers independent of the queue itself
<brixen>
which may be the root of the problem
<brixen>
maybe you have two queues?
<brixen>
a queue of batches and a queue-per-batch ?
<djellemah>
brixen: Nope, only one queue. Well, I'm seeing the other side of that. There *needs* to be an out-of-band mechanism of some kind, within the queue.
<brixen>
in other words, a queue should imply (to me) that there does not exist other information coupling between producers and consumers
<djellemah>
brixen: Igzkaly. So the only reasonable means of communication is the queue itself, but one can't communicate "queue closed" effectively.
<djellemah>
s/Igzkaly/Igzakly/
<brixen>
another way to say this: either all the information to operate exists in *each* item in the queue, or information is shared across multiple items
<brixen>
the latter suggests more than one queue to me
<brixen>
there's not "queue closed" other than an attempt to read returning empty
<[spoiler]>
Not sure what this is about (cba to scroll up, but your api could return a value which means the queue is closed when you try to modify it?
<brixen>
that's what I mean by information coupling outside the items in the queue
<slaught>
djellemah: you have batches which when they are finished you need to flush files? but hte queue is just the list of batch items?
<djellemah>
brixen: I'm using blocking reads.
<djellemah>
[spoiler]: not an api, just the queue and producer/consumer threads.
<djellemah>
slaught: items in the queue are marshalled arrays.
<djellemah>
brixen: information coupling - like the producer knowing there is only one consumer and therefore queueing exactly one poison-token?
<slaught>
djellemah: and the file flushing needs to happen at the end of a batch? and I assume the marshalled array is just teh item to be processed ?
<brixen>
djellemah: yeah, like that
<brixen>
that's a problem
<djellemah>
slaught: yup
<djellemah>
brixen: that's the problem I'm trying to solve.
<slaught>
djellemah: sounds like you are dropping information and trying to re-create it which is causing the coupling.
<brixen>
indeed
<slaught>
optiosn are chucnking the items to processed into a single item in the queue then flush at the end of each chunk. use two queues one for batch items and one for batches to track items processed.
<brixen>
which is why I suggest something like a queue of batches, each of which is a queue of items
<brixen>
yeah
<slaught>
or queue items with information about how many other items are in a batch… but that is messier.
<slaught>
brixen: your suggestion about flushing/committing after N items processed would also work.
<slaught>
the consumer can pull N items of the same type. if there are less than N it only pulls the same type. then flushes/commits after the run.
<slaught>
the is handwaving the failures and recovery but i will assume that problem was already solved. :)
<djellemah>
Yeah, this is all a lot easier if the queue can be closed. That's my point.
<slaught>
djellemah: you can close the queue. you just have to signal everything and track the state.
<slaught>
djellemah: or track the state of the batches whcih is being dropped. eiher way you have to track the state. *shrugs*
<djellemah>
slaught: can't get to Queue's condvar to signal - it's buried in CAPI.
<brixen>
I'm not getting the problem with "closing" the queue
<brixen>
sounds like you're trying to layer this batch information on this "closing" the queue thing
<slaught>
djellemah: just make another condvar
<djellemah>
slaught: needs the mutex, also buried in CAPI.
<brixen>
those are implementation details
<brixen>
irrelevant ones at that
<slaught>
djellemah: I am saying make another mutex
<brixen>
it's like saying, "I can't use this array because the size bits are in the CPU somewhere"
<djellemah>
slaught: two interdependent mutexes is a great way to deadlock. Been there, done that ;-)
Thijsc has quit [Ping timeout: 246 seconds]
<djellemah>
brixen: I'm talking about Queue in ruby, lemme take a look at that wikipedia article...
<slaught>
djellemah: yes that can happen. it is also easy to avoid but using a locking order.
<brixen>
either 1) you have items in a queue, where each consumer can understand the work by dequeuing an item
<brixen>
or 2) you have some other information coupling
<brixen>
djellemah: sorry, gotta run, but I'd suggest getting your head out of Queue and describing your problem in works and state diagrams
<brixen>
s/works/words/
<djellemah>
brixen: ok, ciao. Go has channels. They can be closed.
<slaught>
djellemah: rbx has channels
<djellemah>
slaught: no close
<brixen>
so implement it
<djellemah>
slaught: sorry, not being short, just typing fast.
<brixen>
what is the state of a closed Go channel?
<djellemah>
brixen: Did it for MRI already.
<slaught>
djellemah: didn’t think you were being short.
<brixen>
djellemah: did what? implement closed Channels?
<djellemah>
brixen: Queue#close
<djellemah>
djellemah: it's in the feature requests
<brixen>
class ClosableChannel < Channel ...
<brixen>
djellemah: oh, did you open an issue? I didn't see it
<djellemah>
brixen: yup, it's been quiet for quite a while now. I still need to finish off the tests.
<brixen>
ah well, drop me the # here
<brixen>
bbl...
<djellemah>
djellemah: but also I want to see whether the community accepts it.
diegoviola has joined #rubinius
<djellemah>
slaught: IIRC a few years ago I tried adding a second mutex, ran into deadlocks. I seem to remember getting the ordering right was hard, but I can't remember exactly why now.
<slaught>
djellemah this case it should be straight forward to get the ordering. you have to lock the new mutex before calling the Queue operations. and then unlock it.
<slaught>
then barring bugs or other mutex it should be fine.
<djellemah>
slaught: thinking about it...
<djellemah>
slaught: what happens when 'my' mutex is acquired, and the queue operation blocks? Like when bounded queue is full or the queue is empty?
goyox86 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
djellemah has quit [Quit: Leaving]
pietr0 has quit [Quit: pietr0]
pchalupa has joined #rubinius
<pchalupa>
hi! Could somebody help me to find some information about how rubinius behaves in parallel computing, e.g. memory model, what operations are atomic (I am guessing method/class definition)?
<yorickpeterse>
pchalupa: that's a fairly abstract/generci question, what specifically are you looking for?
<pchalupa>
yorickpeterse hi, sorry I know. I did not have a specific problem in mind, I would like to understand it in general, so I was looking for resources.
<pchalupa>
yorickpeterse I know there are some primitives on Rubinius like Rubinius.lock etc
<yorickpeterse>
pchalupa: well, for one we provide what Ruby in general provides (Thread, Queue, etc)
<yorickpeterse>
those all work using true concurrency, there's no GIL
<pchalupa>
lock is easy to deduce, but Rubinius.memory_barrier is harder to understand which barrier it is
<yorickpeterse>
Atomicity wise, few Ruby classes are atomic
<pchalupa>
yeah stdlib is same, what about method definition, constants etc ...
<yorickpeterse>
method/constant defs are thread-safe since they use a lock I believe
<pchalupa>
e.g. is CONSTANT = 'asd' volatile ?
<pchalupa>
thanks
<yorickpeterse>
Rbx also provides a few other primitives such as Rubinius::Channel
<yorickpeterse>
there's also Rubinius::AtomicReference
<pchalupa>
I definitely appreciate the information but I cannot keep you here for hours to learn all of this:) isn't there a wiki about this stuff? or anything?
<pchalupa>
i guess going throug kernel bootsrup will be also useful
<GitHub29>
[rubinius] chuckremes pushed 1 new commit to ffi-io: http://git.io/RSBZXg
<GitHub29>
rubinius/ffi-io 6330f2c Chuck Remes: remove debug prints again - ps i hate git
GitHub29 has left #rubinius [#rubinius]
<cremes>
if anyone wants to lend me a hand and fix the last failure on IO#gets in spec/ruby/core/io/gets_spec.rb I would be forever grateful. i can’t seem to figure it out.
[spoiler] has quit [Client Quit]
[spoiler] has joined #rubinius
[spoiler] has quit [Client Quit]
[spoiler] has joined #rubinius
[spoiler] has quit [Quit: WeeChat 1.1]
[spoiler] has joined #rubinius
<slaught>
djellemah: your mutex is locked until the queue operation continues.
<slaught>
djellemah: if that is a deal breaker then just use non blocking queue operations. but if your queue can block and that is ok then why is having your mutex block the oerations vs the queue blocking an issue?
<djellemah>
slaught: Here's what I think might happen: say a consumer locks my mutex, then blocks on pop. Now a producer comes along and blocks on my mutex. Producer never gets to push its item, consumer never unblocks.
<yorickpeterse>
djellemah: what are you trying to do?
[spoiler] has quit [Quit: WeeChat 1.1]
<yorickpeterse>
if you're talking about the Mutex class, use Mutex#synchornize vs manually locking/unlocking
<yorickpeterse>
it automatically unlocks upon errors
[spoiler] has joined #rubinius
[spoiler] has quit [Client Quit]
<slaught>
djellemah: given what you have said previously, I htink you will only want your mutex locking on enqueue & close. what does queue.close() mean when the queue is full? are you stopping enqueue’ing or are you stopping all operations. I read it that you wanted to single and end of enqueue operations. then the dequeue would continue until empty.
[spoiler] has joined #rubinius
<slaught>
also you can use non-blocking queue operations and get what you want.
<djellemah>
slaught: my thinking is that once a queue is closed, producers get an exception but consumers will pop successfully until there are no items left. At which point they will no longer block, and always get a fixed value (nil, for example).
<djellemah>
slaught: In the case where there are consumers waiting on an empty queue, close would wake them up and give them the fixed value.
frzng has joined #rubinius
<slaught>
then your implementation would lock your mutex on enqueue and not on dequeue. then close() would permanently lock the mutex.
<slaught>
yorickpeterse: we are talking about a new Queue subclass that uses a Mutex for signaling a queue is closed.
<djellemah>
slaught: wouldn't that just cause all producers to block on enq? I'd want them to get exceptions rather.
<djellemah>
slaught: Actually, in MRI it's easier for close to be part of Queue, and SizedQueue because they rely on the GIL.
<slaught>
djellemah: no. I have been saying mutex but I meant a lock and an instance variable withthe state. the close sets the state permainently to closed. then each producer locks then checks variable, fails or enqueues
<slaught>
djellemah: yes if the program is already single threaded then this problem is trivial.
<djellemah>
slaught: Queue was rewritten a year or two ago (not me) to use the GIL instead of its own mutex. I'll get the ticket number...
goyox86 has joined #rubinius
<|jemc|>
ugh, seriously?
<|jemc|>
they actually moved away from finer-grained locks?
<cremes>
does anyone know how to instantiate a pure Ruby class from rbx’s C++?
<slaught>
djellemah: well I can’t help with mri retardation. but for Rubinius I think the solution will work fine. also adding close to Channel as brixen suggested might be the best overall option then you can just use Channel.
<brixen>
cremes: what do you mean by "pure Ruby class"?
<djellemah>
|jemc|: yup. It kinda makes sense, because there's no point in acquiring two locks if you only need one. And the blocking cases all release the GIL anyway.
<cremes>
brixen: a ruby class that has no C++ backing it. like right now IO is a hybrid… half ruby, half C++.
<brixen>
cremes: generally, you get the constant that references the class and send it new
<brixen>
cremes: but you need a place to get the constant from
<brixen>
cremes: G(object)->get_const might be what you're looking for