<GitHub178>
[jruby] bmulvihill opened pull request #4009: Remove hardcoded base impl (master...allow-str-cmp-override) https://git.io/vKcoH
bbrowning has quit [Ping timeout: 258 seconds]
bbrowning has joined #jruby
rcvalle has quit [Quit: rcvalle]
camlow325 has quit []
bbrowning has quit [Quit: Leaving]
zacts has joined #jruby
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
pawnbox has quit [Remote host closed the connection]
<GitHub21>
[jruby] chrisseaton pushed 3 new commits to master: https://git.io/vKc1q
<GitHub21>
jruby/master 1939d1e Chris Seaton: [Truffle] Add support for a libs field in cext configurations.
<GitHub21>
jruby/master cc7a72f Chris Seaton: [Truffle] Test that we can build and run little cexts using libxml and libssl
<GitHub21>
jruby/master e66b127 Chris Seaton: [Truffle] Test running basic C programs using libxml and openssl.
<headius>
chrisseaton: baby steps towards nokogiri and openssl eh?
<chrisseaton>
Yeah these are testing we can create and run a c extension that depends on a native library, and it will find everything and load it
<flavorjones>
Let me know how I can support libxml support.
<chrisseaton>
Thanks! Do you work on libxml or Nokogiri or something?
<flavorjones>
Nokogiri, yes
<chrisseaton>
Let's see what happens when I try compiling nokogiri now...
<chrisseaton>
Ah yes it uses things from the API we haven't implemented yet, ruby/st.h for example
<chrisseaton>
That stuff makes me nervous... looks like it depends on a lot inside MRI. I guess if rbx can run it we have a hope - their hash is in Ruby so it's like us
bb010g has quit [Quit: Connection closed for inactivity]
Talltree has joined #jruby
<headius>
chrisseaton: if I remember right they believe nokogiri is the cause of a number of crashes, so how well it is supported is unclear
<headius>
I predict you're going to get to know st.h very well over the coming months
<chrisseaton>
I'm going to try running oga next I think, since it was designed to be nice to rbx
<headius>
oga requires a C ext as well
<headius>
probably less invasive though because of rbx
<chrisseaton>
right that's what I meant - a C extension that we might be able to run quickly
<chrisseaton>
chunky works, got a little problem with psd.rb though
<headius>
oh, nice...have you been able to get some idea what the overhead of the truffle cext interface is?
<headius>
I've never seen numbers for rbx but I heard anecdotes about capi overhead being very high
<chrisseaton>
We were 4x faster than MRI with native extensions for TruffleC, but this may be different
<chrisseaton>
I tried running Chunky::Color.r which is tiny, so the overhead stands out, and it was as fast as we run the Ruby code, which is great
<headius>
yeah that seems like a totally different scenario to me
<chrisseaton>
I should check if the generated code is actually the same - that would be good
<headius>
maybe the sulong stuff is able to cheat LLVM instructions as well as TruffleC was able to cheat C
<headius>
hey, one thing that wasn't clear to me
<headius>
can jruby+truffle still support multiple runtimes in the same JVM?
<chrisseaton>
Theoretically but it hasn't been written with that in mind unfortunately - need to back and fix that
<headius>
I ask mostly because of the C ext stuff...sulong would have to simulate separate symbol tables, memory spaces at C level
<chrisseaton>
Lots of statics where they don't really need to be so
<chrisseaton>
It could create multiple heaps, but wouldn't be able to isolate them
<chrisseaton>
But that's undefined behaviour anyway
<headius>
the static state in C exts is probably the biggest reason we gave up on C exts
<headius>
invariably someone eventually wants to deploy this to a server or spin up a second JRuby for something, and kablooey
<chrisseaton>
Sulong itself runs at about 80% of clang -O3 I think
<chrisseaton>
So it does the business on the C code itself
<headius>
hmm
<headius>
so in theory it should be able to rewrite everything to accept passed-in state instead of global state
<chrisseaton>
That's if you're running a standalone C app like an audio encoder or something
<headius>
oh you mean perf wise
<chrisseaton>
Hopefully - there's a lot of loose ends still though
<chrisseaton>
Yeah
<headius>
yeah I would expect sulong should be plenty fast if it's just translating llvm IR to graal
<headius>
the memory/multi-runtime effects are painful
<chrisseaton>
It's a bit higher level than that - it interprets the IR, and uses JIT compilation of the interpreter, like Ruby does
<chrisseaton>
It doesn't directly create Graal nodes
<headius>
the closest we got to that was when Wayne managed to get a C ext to run inside the NestedVM MIPS emulator
<headius>
well yeah but it all goes through graal in the end
<chrisseaton>
So it runs on a stock JVM (except it doesn't as it uses a special NFI at the moment, but we're converting it to JNR or something soon)
<headius>
I just mean not using llvm as the native backend
<chrisseaton>
Yeah
<chrisseaton>
There will be a new paper with more details soon hopefully, and I might submit it as a RubyConf talk (but I'm a bit apprehensive after we got rejected last year)
<headius>
the nestedVM thing was slick because you could load different versions of the same library
<headius>
each runtime could have its own set of loaded C exts
<headius>
but it was veeeery slow :-)
pawnbox has joined #jruby
<headius>
are you familiar with the MVM work?
<chrisseaton>
I think I've heard of it
<chrisseaton>
Shared a code cache and things?
<chrisseaton>
But properly sandboxed so you couldn't tell you were in a shared VM
<headius>
Ruby MVM, not JVM
<headius>
while Tom and I were there, Sun partnered with U of Tokyo to fund _ko1 to work on modifying the C API to support multiple Ruby instances in the same process
<chrisseaton>
ah I was thinking of IBM's multi-tennant VM
<headius>
we served in more of an advisory capacity, but _ko1 and nobu eventually came up with a POC
<headius>
yeah there have been a number of JVM projects
<headius>
several at Sun as well
<headius>
the POC basically isolated global state so it could be swapped out en masse when switching to another in-process VM
<headius>
but it could not run them in parallel because of that
<headius>
mruby's API fixes many things by always passing in VM state
<headius>
we had hoped we might be able to work with ruby-core on adopting a new API that accepts VM state but they're married to the existing API and it will probably never change
pawnbox has quit [Ping timeout: 276 seconds]
<chrisseaton>
Do new cexts use the FFI? Or are people still producing code with the API?
<headius>
vast majority of exts are still using the API
<headius>
and many still use the really nasty stuff like RARRAY and RSTRING macros, st.h, DATA structs, and so on
<chrisseaton>
I'm off
<headius>
chrisseaton: it might be worth asking rbx folks if they have a matrix of known popular C exts and how well they're supported
<headius>
chrisseaton: ttfn
zacts has quit [Ping timeout: 258 seconds]
zacts has joined #jruby
zacts has quit [Quit: WeeChat 1.5]
zacts has joined #jruby
<headius>
zacts: howdy
pawnbox has joined #jruby
<zacts>
hello
<zacts>
:-)
pawnbox has quit [Ping timeout: 244 seconds]
pawnbox has joined #jruby
pawnbox has quit [Ping timeout: 264 seconds]
Hobogrammer has quit [Quit: WeeChat 0.4.2]
pawnbox has joined #jruby
thedarkone2 has quit [Quit: thedarkone2]
yfeldblum has quit [Remote host closed the connection]
zacts has quit [Ping timeout: 260 seconds]
donV has quit [Quit: donV]
pawnbox has quit [Remote host closed the connection]
<GitHub89>
[jruby-openssl] kares closed issue #86: Lookup.loadCertificateOrCRLFile consumes more memory because of frequent memory allocation at Certificate.matches https://git.io/vaE1b
zacts has joined #jruby
yfeldblum has joined #jruby
yfeldblum has quit [Remote host closed the connection]
pawnbox has joined #jruby
yfeldblum has joined #jruby
zacts has quit [Quit: WeeChat 1.5]
skade has joined #jruby
skade has quit [Quit: Computer has gone to sleep.]
zacts has joined #jruby
raeoks has joined #jruby
raeoks has quit [Ping timeout: 240 seconds]
pil-zzZZzz is now known as pilhuhn
e_dub has quit [Read error: Connection reset by peer]
e_dub has joined #jruby
blaxter has joined #jruby
rsim has joined #jruby
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
pawnbox has quit [Ping timeout: 252 seconds]
pawnbox has joined #jruby
pitr-ch has joined #jruby
Specialist has joined #jruby
vtunka has joined #jruby
vtunka has quit [Remote host closed the connection]
vtunka has joined #jruby
zacts has quit [Ping timeout: 250 seconds]
prasun has joined #jruby
yfeldblum has quit [Remote host closed the connection]
zacts has joined #jruby
vtunka has quit [Quit: Leaving]
<GitHub23>
[jruby] kares closed issue #3765: WARNING: SSLSocket#session= is not supported https://git.io/vVLWC
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
blaxter has quit [Read error: Connection reset by peer]
blaxter has joined #jruby
blaxter has quit [Max SendQ exceeded]
blaxter has joined #jruby
pawnbox has quit [Remote host closed the connection]
blaxter has quit [Client Quit]
pawnbox has joined #jruby
yfeldblum has joined #jruby
drbobbeaty has joined #jruby
vtunka has joined #jruby
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
skade has joined #jruby
skade has quit [Client Quit]
pilhuhn is now known as pil-afk
e_dub has quit [Read error: Connection reset by peer]
e_dub has joined #jruby
prasun has quit [Ping timeout: 252 seconds]
skade has joined #jruby
skade has quit [Client Quit]
pil-afk is now known as pilhuhn
bbrowning has joined #jruby
drbobbeaty has joined #jruby
yfeldblum has quit [Ping timeout: 250 seconds]
<GitHub5>
[jruby] chrisseaton pushed 1 new commit to master: https://git.io/vKC2B
<GitHub5>
jruby/master acc3fa2 Chris Seaton: [Truffle] Fix some out-of-date licence information.
<GitHub128>
[jruby] chrisseaton pushed 1 new commit to truffle-head: https://git.io/vKC2E
<GitHub128>
jruby/truffle-head f7ec563 Chris Seaton: Merge branch 'master' into truffle-head
pawnbox has quit [Remote host closed the connection]
vtunka has quit [Quit: Leaving]
e_dub has quit [Quit: ZZZzzz…]
vtunka has joined #jruby
tcrawley-away is now known as tcrawley
pawnbox has joined #jruby
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
pawnbox has quit [Remote host closed the connection]
prasun has joined #jruby
vtunka has quit [Quit: Leaving]
vtunka has joined #jruby
pawnbox has joined #jruby
pawnbox has quit [Ping timeout: 240 seconds]
kubaC has joined #jruby
e_dub has joined #jruby
johnsonch_afk is now known as johnsonch
<flavorjones>
headius and chrisseaton: when rbx was faulting Nokogiri for segfaults, it was primarily around thread behavior during GC; though it appears in later versions of rbx this behavior went away. I haven't heard from the rbx team in quite a while.
<headius>
flavorjones: yeah, none of this is probably relevant anymore since they've changed so much
<headius>
last I heard they still claimed it was nokogiri's fault, but I didn't follow it closely :-)
<flavorjones>
if y'all have suggestions on what to use that would make your life easier, let me know. we're not tied to things like st.h, it was convenient at the time.
<headius>
cool, I'm sure chrisseaton will be chatting with you in the future
<flavorjones>
<chrisseaton>
yeah thanks very much I'll let you know how it goes
pawnbox has joined #jruby
<headius>
flavorjones: what do you know about libxml and thread-safety?
<flavorjones>
enough to be dangerous?
<headius>
hah
<flavorjones>
what do you want to know?
<headius>
I guess I could just look at libxml docs...mostly wondering how it would behave under various concurrent-use scenarios
<headius>
I assume the DOM isn't thread-safe but does it crash? etc
<flavorjones>
generally, one-thread-per-doc is the model. operations on a single document need to happen within a single thread -- they're not threadsafe. but you can have multiple threads, each operating on a separate document.
pawnbox has quit [Ping timeout: 244 seconds]
<flavorjones>
it tends to get weird when you consider that each doc has a "dictionary" of shared strings, and moving nodes between docs without Nokogiri's safety checks (copy-on-move) will totally end up segfaulting, particularly at GC time
<headius>
lovely
<flavorjones>
am I making sense?
<headius>
yeah
<flavorjones>
end of the day, it's mostly all re-entrant C, but the data structures aren't threadsafe.
<headius>
I hope the cext stuff doesn't stumble over thread-unsafe exts and libs too much
<flavorjones>
I'm totally enthralled with the TruffleC work, but I have no idea how it works. I'd totally attend a talk on it, FWIW
camlow325 has quit [Read error: Connection reset by peer]
vtunka has joined #jruby
camlow325 has joined #jruby
camlow32_ has joined #jruby
camlow325 has quit [Read error: Connection reset by peer]
camlow325 has joined #jruby
camlow32_ has quit [Read error: Connection reset by peer]
<headius>
flavorjones: TruffleC is the old project and it sounds like it's not going anywhere now...it was the interpreter for C written in Truffle
<headius>
the new project is using Sulon, an impl of LLVM bitcode in Truffle
<headius>
Sulong
camlow325 has quit [Read error: Connection reset by peer]
camlow325 has joined #jruby
<headius>
early numberrs sound ok...perf about like pure-Ruby on JRuby +Truffle it sounded like
pitr-ch_ has joined #jruby
pitr-ch has quit [Read error: Connection reset by peer]
<GitHub138>
[jruby] headius opened pull request #4010: Reopen stdio streams when run from Drip, to pick up its FIFOs. (master...reopen_stdio_for_drip) https://git.io/vKCAR
bbrowning has quit [Ping timeout: 260 seconds]
pawnbox has joined #jruby
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
e_dub has quit [Ping timeout: 240 seconds]
bga57 has joined #jruby
<chrisseaton>
flavorjones: the real win is inking and constant folding between the two languages
<chrisseaton>
And inline caching for calls from C back into Ruby
<chrisseaton>
Inlining, not inking
etehtsea has joined #jruby
bbrowning has joined #jruby
etehtsea has quit [Client Quit]
* lopex
was about to ask
zacts has quit [Quit: WeeChat 1.5]
<headius>
chrisseaton: how does inline cache work? you do some patching of rb_funcall and friends?
<chrisseaton>
rb_funcall uses a normal Ruby #send call so has normal caching
<chrisseaton>
Look in cext.rb
<headius>
'normal caching'?
<headius>
is inline caching normal or not? :-)
<headius>
does this specialize for everywhere rb_funcall is called as well?
etehtsea has joined #jruby
e_dub has joined #jruby
<etehtsea>
is -Xerrno.backtrace=true affecting performance?
<headius>
etehtsea: it certainly could if you have a lot of nonblocking IO
etehtsea has quit [Quit: Computer has gone to sleep.]
<chrisseaton>
headius: we have essentially one mechanism to cache method calls and constant lookups, and it's the same code used everywhere, that's what I mean by normal
<headius>
ok
Specialist has quit [Remote host closed the connection]
<chrisseaton>
and yes that cache is pushed down the call stack, so you could have one cache for each place you call rb_funcall, or if it makes sense it could be one cache for each place which will then call rb_funcall
<chrisseaton>
avoids the caches being megamorphic
pilhuhn is now known as pil-afk
<chrisseaton>
and lets you add extra levels of abstraction without ruining the caches
<GitHub112>
jruby/master d9dd559 Charles Oliver Nutter: Hook up instance var copying and link registration for Regexp....
<headius>
chrisseaton: does truffle have any inlining budgets yet?
tjohnson has joined #jruby
<chrisseaton>
yes, I'm not familiar with it myself, but it's to do with things like how polymorphic the code is, how many graal nodes, whether the callee uses the caller frame, and other things like that
<chrisseaton>
I talk about this a bit in the latest VMSS talk if you watched it
<headius>
I have not
<chrisseaton>
it's on /r/ruby atm
bbrowning_ has joined #jruby
<chrisseaton>
I haven't studied it in depth because it always seems to do pretty much I'd like
<chrisseaton>
One day I'll probably have to learn to tune it
bbrowning has quit [Ping timeout: 258 seconds]
bbrowning_ is now known as bbrowning
<cprice404>
is there a target date for 1.7.26?
kith has joined #jruby
<GitHub169>
[jruby] estolfo opened issue #4011: Performance decrease / memory usage increase between 1.7 and 9.1 https://git.io/vKWqq
subbu|meeting is now known as subbu|afk
rsim1 has joined #jruby
<headius>
chrisseaton: I guess I was wondering more about code size...even with the less-specialized optimizations we get out of hotspot, we can easily fill up the code cache
<chrisseaton>
I haven't seen that happen myself yet, so I'll start to think about when I see it
<chrisseaton>
But yeah, we trade off memory for peak performance, that's the deal we're making
<headius>
we've had people have to bump code cache up hundreds of MB just for hotspot
<headius>
ah
<headius>
well, that kinda answers that then
<chrisseaton>
Tenderlove was joking about this the other day, saying you can't get memory out of thin air, but say you want to reduce the time per request for each customer. How can you do that? You can't add more sequential processing power to a single core. You can't automatically parallelise because we don't know how to do that. What you can do is add an extra 8 GB of
<chrisseaton>
RAM to your server. It's the only degree of freedom we have at the moment.
rsim has quit [Ping timeout: 252 seconds]
pawnbox has quit [Remote host closed the connection]
<chrisseaton>
Personally I think our memory overhead and warmup time is going to be reasonable in the long term, but I don't have any evidence so there's no point arguing about it
<headius>
chrisseaton: good talk, wish there had been time for more discussion
<chrisseaton>
There was a lot of talking afterwards but we didn't film it to allow students to ask questions without feeling embarassed
<GitHub20>
[jruby] estolfo opened issue #4012: Performance decrease / memory usage increase between 1.7 and 9.1 https://git.io/vKWq4
<headius>
the sad thing about your talks is that if the JVM would duplicate method bodies and have proper EA a lot of this would be moot
<headius>
so we have to do it ourselves
<chrisseaton>
Have you started a branch to try that manually splitting methods by literally copying the byte code idea that you had?
<headius>
oh sure, I've done the experiment in a few ways and it works
<headius>
memory use explodes
<headius>
our hands are kinda tied because we have users and can't just suddenly tell them to allocate another 4GB to the JVM
subbu|afk is now known as subbu
<chrisseaton>
Sure
<headius>
if we could, I could make indy call sites duplicate any number of target methods
<GitHub103>
[jruby] estolfo closed issue #4011: Performance decrease / memory usage increase between 1.7 and 9.1 https://git.io/vKWqq
camlow32_ has quit [Remote host closed the connection]
camlow325 has joined #jruby
yfeldblum has joined #jruby
pitr-ch_ has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
bbrowning_away is now known as bbrowning
pitr-ch has joined #jruby
skade has quit [Ping timeout: 250 seconds]
subbu is now known as subbu|lunch
skade has joined #jruby
pawnbox has quit [Read error: Connection timed out]
pawnbox has joined #jruby
pitr-ch has quit [Ping timeout: 240 seconds]
e_dub has quit [Ping timeout: 276 seconds]
pitr-ch has joined #jruby
pawnbox has quit [Read error: Connection timed out]
rsim has joined #jruby
rsim1 has quit [Ping timeout: 252 seconds]
pawnbox has joined #jruby
bbrowning has quit [Ping timeout: 258 seconds]
bbrowning has joined #jruby
donV has joined #jruby
<donV>
Hi all!
skade has quit [Read error: Connection reset by peer]
skade has joined #jruby
e_dub has joined #jruby
bbrowning_ has joined #jruby
bga57 has quit [Ping timeout: 250 seconds]
bbrowning has quit [Ping timeout: 276 seconds]
bga57 has joined #jruby
subbu|lunch is now known as subbu
skade has quit [Read error: Connection reset by peer]
bbrowning_ is now known as bbrowning
skade has joined #jruby
e_dub has quit [Quit: ZZZzzz…]
yfeldblum has quit [Ping timeout: 250 seconds]
prasunanand has quit [Ping timeout: 246 seconds]
zacts has joined #jruby
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
e_dub has joined #jruby
rcvalle has joined #jruby
rsim has quit [Quit: Leaving.]
yfeldblum has joined #jruby
e_dub has quit [Quit: ZZZzzz…]
skade has quit [Read error: Connection reset by peer]
skade has joined #jruby
thedarkone2 has quit [Read error: Connection reset by peer]
<headius>
heh, well first POC of specializing everything proved...interesting
<headius>
4x slower on red/black and only used 3GB of memory
<headius>
oh oops
<headius>
maybe I shouldn't make it rejit every method every time they're called
thedarkone2 has joined #jruby
e_dub has joined #jruby
zacts has quit [Ping timeout: 244 seconds]
<headius>
that's better
camlow325 has quit [Ping timeout: 252 seconds]
e_dub has quit [Quit: ZZZzzz…]
skade has quit [Read error: Connection reset by peer]
bbrowning is now known as bbrowning_away
skade has joined #jruby
zacts has joined #jruby
skade has quit [Read error: Connection reset by peer]
<headius>
well red/black is a bit faster already but not a ton
skade has joined #jruby
tcrawley is now known as tcrawley-away
johnsonch is now known as johnsonch_afk
camlow325 has joined #jruby
<headius>
promising
camlow325 has quit [Remote host closed the connection]
pawnbox has quit [Ping timeout: 260 seconds]
skade has quit [Read error: Connection reset by peer]
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<headius>
chrisseaton: do you know of any good docs on tuning graal?
<chrisseaton>
On what all the options do? No I don't know much about that, sorry. But if you mean more generally, then IGV will help a lot.
<headius>
oh there's -G:+PrintFlags
<headius>
hmm
<headius>
that doesn't seem to do anything in the oracle builds
<chrisseaton>
The options are all system properties now
<chrisseaton>
-Dgraal. and -Djvmci. I think
<chrisseaton>
The options are freaking complicated at the moment, and some are different between JDK 8 and 9
camlow325 has joined #jruby
camlow325 has quit [Remote host closed the connection]
donV has quit [Quit: donV]
camlow325 has joined #jruby
camlow32_ has joined #jruby
camlow325 has quit [Read error: Connection reset by peer]
pawnbox has joined #jruby
camlow32_ has quit [Read error: Connection reset by peer]
camlow325 has joined #jruby
camlow325 has quit [Read error: Connection reset by peer]
camlow32_ has joined #jruby
enebo has quit [Quit: enebo]
camlow32_ has quit [Read error: Connection reset by peer]
camlow325 has joined #jruby
<Talltree>
i bet you get that really often, but the puma guide is recommending to switch to real threaded ruby implementations like jruby for performence reasons, but how much is the inprovement against a standard ruby version?
pitr-ch has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<Talltree>
thats really true? from what i'm reading on there it nearly doubles the req/s
<Talltree>
1366.99-> 2588.18 rails
subbu is now known as subbu|away
yfeldblum has quit [Read error: Connection reset by peer]
zacts has quit [Ping timeout: 272 seconds]
yfeldblum has joined #jruby
<headius>
chrisseaton: oh, that's unfortunate
<headius>
I need to get more info out of graal
<headius>
I'll try IGV again...when I first played with it like six years ago it was really painful to use
<chrisseaton>
It's often a bit of a forest for truffle code, but for reasonably sized Java methods where each node is a bytecode to start with its easier
<headius>
yeah that wasn't as bad...the tool was still terrible though
<headius>
it was more about the tool than the graph
<headius>
going to be playing with more specialization and getting local caching into the java code
<headius>
getitng all ruby methods to specialize to call site was easier than I expected
zacts has joined #jruby
<headius>
should be possible to do the same for java methods by loading multiple copies of the bytecode
<headius>
then we can optimize core classes just like ruby
zacts has quit [Client Quit]
<headius>
hey, there we go...getting red/black below 0.4s
<headius>
calling it a night...ttfn
__hudson__ has quit [Ping timeout: 240 seconds]
camlow32_ has joined #jruby
yfeldblum has quit [Remote host closed the connection]