_djbkd has quit [Remote host closed the connection]
mrmargolis has joined #jruby
yokolet has joined #jruby
camlow325 has quit []
e_dub has quit [Quit: Leaving]
dinfuehr has joined #jruby
e_dub has joined #jruby
havenwood has quit [Ping timeout: 264 seconds]
dinfuehr has quit [Ping timeout: 244 seconds]
Locke23rus has quit [Remote host closed the connection]
yfeldblum has quit [Ping timeout: 256 seconds]
iamjarvo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Hobogrammer has joined #jruby
nirvdrum has quit [Ping timeout: 246 seconds]
mrmargolis has quit [Remote host closed the connection]
havenwood has joined #jruby
iamjarvo has joined #jruby
iamjarvo has quit [Max SendQ exceeded]
iamjarvo has joined #jruby
iamjarvo has quit [Max SendQ exceeded]
iamjarvo has joined #jruby
subbu has joined #jruby
mrmargolis has joined #jruby
dinfuehr has joined #jruby
dinfuehr has quit [Ping timeout: 264 seconds]
subbu has quit [Quit: Leaving]
deobalds has joined #jruby
iamjarvo_ has joined #jruby
iamjarvo_ has quit [Max SendQ exceeded]
iamjarvo_ has joined #jruby
iamjarvo has quit [Ping timeout: 265 seconds]
iamjarvo_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
simi has quit [Ping timeout: 272 seconds]
vikaton has quit []
<Antiarc>
The new mongodb driver calls Process.pid on every message to make sure the host process hasn't changed (sigh), which has substantial overhead on JRuby due to the whole JNR/FFI thing. Is there a faster way to check this that I could suggest?
yokolet has quit []
mrmargolis has quit [Remote host closed the connection]
<JRubyGithub>
[jruby] enebo pushed 1 new commit to master: http://git.io/vIfn4
<JRubyGithub>
jruby/master 4248eab Thomas E. Enebo: Combine these two tests so we are not hard-coding our defaults in a test file and failing when we decide to change defaults
JRubyGithub has left #jruby [#jruby]
<bbrowning>
imperator: if you're looking at the histogram, look at Total Size column
<bbrowning>
with an OOM situation there is quite often something fishy in the first 10-20 rows there
<bbrowning>
ie with my leak the top item was 300MB of int arrays
<bbrowning>
represented in jhat as class [I
tcrawley-away is now known as tcrawley
donV has joined #jruby
<imperator>
class [B and class [C are top 2
<imperator>
followed by class [Ljava.lang.Object; and class [Ljava.lang.String;
<bbrowning>
that's pretty typical so far
<bbrowning>
byte and char arrays followed by objects and strings
<imperator>
first jruby instance is class [Lorg.jruby.ir.operands.Operand; at fifth most
<imperator>
s/instance/mention
<bbrowning>
what's the instance count there?
<bbrowning>
my leak had a high Operand count as well
<bbrowning>
139k
<imperator>
149273 - 4011024
<imperator>
count - size
donV has quit [Client Quit]
<bbrowning>
enebo: do you know what a reasonable count of Operands would be? would we expect them to be hundreds of thousands in a typical small app?
<imperator>
well, this is a large-ish rails app
<bbrowning>
149k I'd guess is one of the highest # of instances of any class in imperator's heap
<bbrowning>
imperator: if you look at the query "Show instance counts for all classes (including platform)" from the jhat home page, does it show Operand near the top there?
<enebo>
bbrowning: operands are the most common way we represent data in IR so tons of operands is normal
<bbrowning>
imperator: well my app was quite small and still had tons of operands
<bbrowning>
hmm
<imperator>
bbrowning, yep, operand is 2nd
<bbrowning>
probably only behind string
<imperator>
right
<bbrowning>
enebo: but holy cow this many? :)
<enebo>
a = 1+1 is 4+ operands
<bbrowning>
Operands take up more memory in my heaps than Strings ;)
<enebo>
bbrowning: but with that said I did spend time making common operands share like Variable
<bbrowning>
imperator: so in your case Operands are only taking up 4MB of your 1024MB heap
<enebo>
bbrowning: oh you mean Operand[] too though right?
<bbrowning>
imperator: is there something with a total size of many hundreds of MB?
<enebo>
Every instr is backed by a single Operand[]
<bbrowning>
enebo: class [Lorg.jruby.ir.operands.Operand;
<bbrowning>
so yeah I guess that is an array of them?
<enebo>
bbrowning: yeah
<enebo>
so 1 array per instr instance
<bbrowning>
imperator: one of the 4 above your operand - byte, char, object, or string arrays
<bbrowning>
one of those must be taking up a huge amount of heap
<enebo>
bbrowning: actually one issue we have is we duplicate every instr when we JIT (or full build in -X-C) so we can end up with 2x as many instrs
<enebo>
bbrowning: we keep both copies because we want to be able to deopt
<imperator>
char 106130 - 6674192
<bbrowning>
imperator: that's still only 6MB :/
<imperator>
object 94929 - 6039136
<bbrowning>
a long way away from blowing up a 1024MB heap
<enebo>
bbrowning: if we get rid of ipc/rpc info from instr we can prevent most of our instr dup’ing and it will help but we JIT too much atm too so either fix will reduce memory
<imperator>
yeah, this must be something else
<imperator>
i need to do some task inspection i think
<imperator>
i also sort of blindly replaced thin with puma as a gem dep just trying to get it to work
<imperator>
but, as far as i can tell this task is just doing a drop/create/migrate
<bbrowning>
imperator: it's odd that the heap dumped on OOM if there's nothing big in the heap :/
<bbrowning>
I'd expect to see something at the top of that heap histogram with hundreds of MB
<JRubyGithub>
[jruby] jtulach opened pull request #3017: Adopting to Redesigned Truffle Inter-operability API (truffle-head...NewInterOpAPI) http://git.io/vIf9X
JRubyGithub has left #jruby [#jruby]
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] nirvdrum pushed 2 new commits to master: http://git.io/vIfQO
<JRubyGithub>
jruby/master e0668e3 Kevin Menard: [Truffle] Fixed an NPE when evaluating empty interpolation (#{}).
<JRubyGithub>
jruby/master ee6b66c Kevin Menard: [Truffle] All Regexp#encoding specs now passing.
JRubyGithub has left #jruby [#jruby]
x1337807x has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
iamjarvo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
drguildo has left #jruby [#jruby]
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] nirvdrum pushed 1 new commit to master: http://git.io/vIfdS
<JRubyGithub>
jruby/master 361d393 Kevin Menard: [Truffle] All Regexp#compile specs are now passing.
JRubyGithub has left #jruby [#jruby]
<nirvdrum>
enebo: It looks like there are specs that verify the Regexp#source encoding. I'll see if I can find a cheap way to handle it.
mcclurmc has quit [Remote host closed the connection]
mcclurmc has joined #jruby
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] pitr-ch pushed 18 new commits to master: http://git.io/vIfpq
<JRubyGithub>
jruby/master 8156e58 Petr Chalupa: [Truffle] BigDecimal: add special value handling to :/, :quo
<JRubyGithub>
jruby/master c3cc29e Petr Chalupa: [Truffle] BigDecimal: add :exponent
<JRubyGithub>
jruby/master 0398913 Petr Chalupa: [Truffle] BigDecimal: add :abs
JRubyGithub has left #jruby [#jruby]
vtunka has quit [Quit: Leaving]
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] pitr-ch closed pull request #3012: [Truffle] BigDecimal more features (master...master) http://git.io/vk9zx
JRubyGithub has left #jruby [#jruby]
phlebas has quit [Ping timeout: 276 seconds]
bb010g has quit [Quit: Connection closed for inactivity]
iamjarvo has joined #jruby
iamjarvo has quit [Max SendQ exceeded]
temporalfox has joined #jruby
mister_solo has quit [Ping timeout: 245 seconds]
temporal_ has quit [Ping timeout: 276 seconds]
<enebo>
nirvdrum: cool
x1337807x has joined #jruby
iamjarvo has joined #jruby
iamjarvo has quit [Max SendQ exceeded]
iamjarvo has joined #jruby
erikhatcher has joined #jruby
<nirvdrum>
enebo: Do you guys have any helpers for running rubyspecs? Or do you just run mspec directly?
iamjarvo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<enebo>
nirvdrum: I typeically just run it directly and without tags
<bbrowning>
hmm maybe I can simplify this a bit more and create a much smaller in-memory .po file
<imperator>
nirvdrum, he has very kindly been helping me
<bbrowning>
for a self-contained reprocuder
baroquebobcat has quit [Quit: baroquebobcat]
<nirvdrum>
imperator: Are you djberg96?
<imperator>
yup
baroquebobcat has joined #jruby
<nirvdrum>
Funky. I just came across one of your win32 gems a couple days ago.
<nirvdrum>
I thought it'd be helpful to get some of it into jnr-posix. But I didn't know what the Artistic 2.0 license entailed and whether it was compatible, so I left for the time being.
<imperator>
oh? any issues, lemme know, i'll be happy to help
<nirvdrum>
Small world.
<imperator>
ah, yeah, some people don't like that license
<imperator>
it's fairly permissive
<imperator>
the stuff i've written for RH is Apache 2.0
<nirvdrum>
I think RedHat lawyers just combed jnr-posix recently. Corporate lawyers aren't fond of anythin off the beaten path. Not really your problem. Like I said, it just needs to be looked at.
<nirvdrum>
So you work for RH, too?
skade has quit [Ping timeout: 265 seconds]
iamjarvo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<enebo>
bbrowning: went for bike ride after eating
<enebo>
bbrowning: I wonder if we have temp vars retaining intermediate copies of large objects in some way…still does not seem like it would explain it
<bbrowning>
this script ends up with a RubyArray taking up 97% of the heap
<enebo>
bbrowning: I wonder if that lib actually is working with smaller values
<enebo>
e.g. we have some other bug and this is not even acting correct
<bbrowning>
this string is quite a bit smaller than the real-world one imperator's app was blowing up on
<enebo>
yeah
<bbrowning>
the actual results of the @q array look identical b/w 1.7.20 and 9k with just a spot-check
<enebo>
ok
<enebo>
It would have been nicer had that been the case :)
<bbrowning>
so in 9k this @q array is a 406MB RubyArray before it OOMs with my 512MB heap
<bbrowning>
4,618 entries in it
<bbrowning>
I'd expect 10k entries if it ran without OOMing
<bbrowning>
each entry is another RubyArray with 2 values
<bbrowning>
the first value is a symbol as we'd expect - :MSGID or :STRING
<bbrowning>
the 2nd value is the problematic one
<enebo>
douns like correctness problem to me
<bbrowning>
2nd value is a RubyString with a bytelist containing the bytes of the entire file
<enebo>
$& or $1 maybe are not correct values while running but produces the right result?
<bbrowning>
it looks like the line "@q.push [:STRING, $1]" is pushing a ByteList with an offset and length into the array
<bbrowning>
the ByteList contains the bytes of the entire string, but uses the offset and length to only "store" the right part
<bbrowning>
so every time we push a string into this array we're pushing a RubyString backed by a ByteList containing a MUCH larger byte[] with an offset and length into a tiny bit of that byte[]
<enebo>
hmmm but looking at nth_match it apepars we makeShared19 on it
<enebo>
since 2013
<enebo>
bbrowning: and if they are sharing the same big string then it does not matter
dinfuehr has joined #jruby
<enebo>
1.7 appears to also makeShared
pjammer has joined #jruby
<bbrowning>
enebo: all I can say is in the heap different RubyStrings are backed by different ByteLists
<bbrowning>
each ByteList containing a ton of text and an index/length
<enebo>
perhaps some indirect way we are making a non-shared bytelist and then backing that
<enebo>
Have I mentioned how much I hate COW?
<imperator>
how much?
<enebo>
It seems like such a clever way of saving space until you have to debug it :)
<enebo>
imperator: 6'
x1337807x has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<imperator>
many feets worth of hate, got it
<enebo>
So I would guess something is making a new string without continuing to share it but the regexp nth ref is sharing that so we get n massive backing bytelists. Sound about right bbrowning ?
<bbrowning>
enebo: yessir
<bbrowning>
I'm still trying to work on an even simpler reproducer to really understand how it's triggered
<enebo>
I would try converting the case/when to if/elsif
<bbrowning>
that one change can get it running in a 64mb heap on 9k
<bbrowning>
imperator: so as a workaround you can strip all newlines out of your .po files perhaps? heh
<imperator>
hah, dunno
<enebo>
wow
<enebo>
bbrowning: sounds like we are matching too much then
<bbrowning>
enebo: but remember the part that ultimately gets captured, $1, is "correct"
<bbrowning>
correct in the sense that the RubyString displays correctly - incorrect in that it's backed by a gigantic ByteList where most of those bytes are wasted
<enebo>
yeah
<enebo>
nirvdrum: is it possible we changed COW and made something shared no longer shared with regexps?
<enebo>
bbrowning: It sounds like it describes the observed behavior well enough
subbu|lunch is now known as subbu
<nirvdrum>
enebo: I don't think I touched anything related to that. But doing a checkout from 3 days ago would be sufficient to check.
<enebo>
bbrowning: but we would need to copy it as a whole valid substring
<nirvdrum>
I haven't been working on this too long.
<enebo>
nirvdrum: yeah I mean we have had these divergent so long I am not sure we can know if we might have earlier too
<enebo>
bbrowning: if we were non sharing something smaller it would become the smaller string
<enebo>
bbrowning: in that snnd $’ is most likely to be bad guy
<nirvdrum>
This is the first I've really looked at JRuby regexp. I obviously did a lot of String work, but I don't think anything to affect CoW.
<nirvdrum>
Maybe something in joni? I can't tell what version 1.7.20 is using.
<bbrowning>
enebo: I can eliminate $' and still get the error
<enebo>
bbrowning: cool…I will stop looking at that then :)
<bbrowning>
yeah I dunno - worth filing a bug, at least :)
* bbrowning
does that
<nirvdrum>
enebo: FWIW, this doesn't blow up with Truffle and we're using very similar matching logic.
<enebo>
nirvdrum: ok good to know. I will concentrate on this once I get parsing green again
<nirvdrum>
I'll take a quick poke myself. It might be instructive.
<nirvdrum>
All of Regexp has been trial by fire for me thus far.
<enebo>
nirvdrum: we have had these errors in the past as well and I sort of wish we had some extra heuristiuc like don’t share off anything bigger than 1000 bytes
<enebo>
which would hurt some people but perhaps greatly reduce COW weirdness
iamjarvo has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<imperator>
perhaps look to other languages for guidance? what do they do?
<enebo>
yeah at some point we should revisit our impl.
<enebo>
our COW impl is largely modelled after MRIs
<JRubyGithub>
[jruby] bbrowning opened issue #3019: Small string values backed by HUGE non-shared ByteList leading to large memory usage under 9k http://git.io/vIU2k
JRubyGithub has left #jruby [#jruby]
<enebo>
bbrowning: sounds fine
<enebo>
bbrowning: thanks for simplifying this
iamjarvo has joined #jruby
iamjarvo has quit [Max SendQ exceeded]
<nirvdrum>
enebo: FWIW, we don't do any CoW with our Strings yet.
<bbrowning>
enebo: no problem - not sure if this is some obscure edge case or will be a common memory usage bomb under 9k
iamjarvo has joined #jruby
iamjarvo has quit [Max SendQ exceeded]
<enebo>
bbrowning: I sure hope this is just a simple mistake easily corrected
<enebo>
bbrowning: nothing weird about this pattern of code
iamjarvo has joined #jruby
<imperator>
forgive my ignorance, but does CoW negatively affect performance in multithreaded code?
<imperator>
btw, thank you very much bbrowning enebo and nirvdum - this has been interesting
<imperator>
didn't realize how far the rabbit hole went when i started this
<bbrowning>
imperator: jruby 9k is bound to run into a couple of interesting bugs like this :)
<nirvdrum>
imperator: I haven't benchmarked, but it shouldn't be too bad. The first thread mutating the object makes the copy and all other threads work with that new copy. You'll need to lock of course.
dinfuehr has quit [Remote host closed the connection]
dinfuehr has joined #jruby
<enebo>
imperator: well if you modify a string you need to make a copy so offthread mutation will just copy the first copy down to what it needs
<enebo>
imperator: original thread won’t realize it and if it needs to mutate it will copy as well
<enebo>
imperator: then original byte[] will just GC
<enebo>
imperator: unless something else is retaining a reference
<nirvdrum>
I guess that's the case where you don't lock :-)
<enebo>
yeah the backing array is immutable itself if shared
<enebo>
ok I was thining why is it deleting data :)
<enebo>
bbrowning: you try this on MRI?
<nirvdrum>
enebo: Wouldn't freeze always return the same object though?
<bbrowning>
enebo: imperator ran the original real app on MRI and didn't see any large memory usage
<enebo>
I don’t see the value of this override
<enebo>
I am guessing it was to kill the COW
<enebo>
err copy
<enebo>
make a copy
<bbrowning>
which I'm fine with, but if you do that then also prune all the extra crap out of the bytelist ;)
<enebo>
if (OBJ_FROZEN(str)) return str;
<enebo>
rb_str_resize(str, RSTRING_LEN(str));
<enebo>
return rb_obj_freeze(str);
<enebo>
So it is the same
<enebo>
bur perhaps they freeze their map str differently
<nirvdrum>
It looks like the size should be set before the modify() call. Of course, it's not that easy.
<enebo>
match
<nirvdrum>
But the dup call is working with the original size.
<nirvdrum>
Actually, it's using the same value. So I don't think that's quite it either.
tcrawley is now known as tcrawley-away
<enebo>
rb_str_resize will not resize if they are the same size
<enebo>
but it also has a lot of cases
<nirvdrum>
I think the resize is a red herring.
<nirvdrum>
Looking in a debugger, the real size and the byte[] size are the the same +/- a byte from what I can tell.
<nirvdrum>
But the modify() call is causing twice as many of these buffers to be allocated due to all the ByteList duping.
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] enebo closed issue #3015: Syntax Error on nested hash that parses with MRI 2.0 but not with JRuby 9000 http://git.io/vkNCH
JRubyGithub has left #jruby [#jruby]
<bbrowning>
nirvdrum: looking in my heap dumps I have ByteLists with a realSize of 5 but a byte[] size of 128,819
<bbrowning>
each one is from this match data
<bbrowning>
and the addition of the new RubyString#freeze would explain why 1.7.20 and 9k differ here
<bbrowning>
guess I can just comment out resize locally to verify
<enebo>
bbrowning: you can also comment out freeze and verify
<bbrowning>
they are essentially the same thing, so I will just comment out the whole override
<bbrowning>
the original commit surely exists for a reason so this probably isn't the right fix
<nirvdrum>
bbrowning: I'm not looking at a heap dump. But I set a break point for when real size and the byte array didn't match and the one time it happened, it was off by 1 byte.
<bbrowning>
nirvdrum: and that was when running against my test script?
rsim has joined #jruby
<nirvdrum>
Yeap. And then I ran out of memory.
<bbrowning>
hrm
<bbrowning>
yeah commenting out freeze doesn't fix things :/
<nirvdrum>
Same here.
<bbrowning>
nirvdrum: so you're on to something there - there is some point where realSize and the byte size differ by a lot
<lopex>
enebo: some String bang methods might remain the string shared
<enebo>
lopex: original script did not use String#[]
<nirvdrum>
enebo: Good point. I quickly glanced at it.
<enebo>
lopex: yeah but it does not really matter does it? all actions after that strp! whether it is still sharing or not may share as new strings
<enebo>
lopex: this doesn ot recursively call this parse method
<enebo>
I actually think bbrowning_away explanation of resize completely explains the behavior
<nirvdrum>
So I guess the real question is was = somehow smart enough to view into a string in 1.7.20? I somehow doubt that.
<enebo>
our resize assumes you will stop sharing
<nirvdrum>
enebo: I was still seeing the problem after I removed both the strDup and the freeze call, however.
<lopex>
enebo: so it's ultimately modify/modifyCheck issue ?
<enebo>
we should just perhaps make subset resizes not stop sharing and then we can match MRI logic but that would be the same to me as just removing it since this is only for a reduced backing store change
<enebo>
nirvdrum: oh
<enebo>
nirvdrum: so that new freeze being removed does not fix the issue
<nirvdrum>
Correct.
<enebo>
nirvdrum: ah I was not paying attention :)
<nirvdrum>
Where I'm seeing the leak is Regexp.nth_match calls makeShared19. That's causing what should by a 5 byte string to keep a 120K ByteList around.
<nirvdrum>
s/should by/should be/
<lopex>
is it ultimately the new freeze thingy issues ?
<nirvdrum>
And the result is retained in @q.
<nirvdrum>
That seems to support the large values he was seeing in the heap dump, too.
JRubyGithub has joined #jruby
JRubyGithub has left #jruby [#jruby]
<JRubyGithub>
jruby/master 5f93f4c Thomas E. Enebo: Fixes #2744. Syntax error with file that has OK syntax with MRI 2.2
<JRubyGithub>
[jruby] enebo pushed 1 new commit to master: http://git.io/vITfk
<nirvdrum>
But, this code doesn't look like it's changed in 6 or 7 years, so I'm not sure why it doesn't blow up in 1.7.20.
<enebo>
yeah this has to be something deciding to stop sharing a larger view and really copy the backing byte array
<lopex>
like what ?
<lopex>
there's not thresholds on that :)
<enebo>
lopex: originally I suspected $’ was not sharing and just making a smaller one with new byte[] and then $1 was sharing that but ben figured out that was not the case
<lopex>
oh the $ ones mght be the leak definitely
<lopex>
I need to move where your time zone is
<lopex>
it's always midnight when some on irc comes up here
<lopex>
something
<enebo>
lopex: it would be double cool if you have also been drinking
SuperTaz_ has quit [Ping timeout: 248 seconds]
<lopex>
enebo: I only drink on weekends if only
<lopex>
enebo: last weekend we drank all that american beer with my brothers
<nirvdrum>
enebo: I still think in the Regexp.nth_match case, it makes sense to extract a copy of the string rather than share it. Maybe you could use a heuristic to determine.
<lopex>
enebo: what the status of those $ now ?
<lopex>
thread locals or scoped ?
<lopex>
I remember $ were an endless nightmare
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] bjfish pushed 1 new commit to master: http://git.io/vITUz