arturaz has quit [Remote host closed the connection]
nateberkopec has joined #jruby
goyox86 has joined #jruby
cristianrasch has quit [Quit: Leaving]
nateberkopec has quit [Quit: Leaving...]
Hobogrammer has joined #jruby
Silent__ has quit [Ping timeout: 250 seconds]
yfeldblu_ has quit [Ping timeout: 248 seconds]
bbrowning_ is now known as bbrowning
cristianrasch has joined #jruby
nateberkopec has joined #jruby
nateberkopec has quit [Client Quit]
shellac has quit [Quit: Computer has gone to sleep.]
nirvdrum has joined #jruby
nateberkopec has joined #jruby
samphippen has quit [Read error: Connection reset by peer]
fkr has joined #jruby
<fkr>
hi
<fkr>
in a piece of code, we're doing: data['content_length'].to_i - if this is beyond the scope of an int, this gives an exception if run in jruby ("(RangeError) too big for int:").
<fkr>
am not a ruby person, just trying to fix this code and was wondering, since in ruby .to_i should do bignum automatically...
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] eregon pushed 1 new commit to master: http://git.io/vmrNu
<JRubyGithub>
jruby/master 431c4f4 Benoit Daloze: [Truffle] JT: bash needs extra quoting to pass args as-is.
JRubyGithub has left #jruby [#jruby]
tcrawley-away is now known as tcrawley
<nirvdrum>
fkr: Which version of JRuby?
subbu has joined #jruby
<nirvdrum>
fkr: I can't reproduce on 1.7.20 or on master.
<nirvdrum>
Are you using the produced value as an index for something?
<fkr>
yeah, within an irb on 1.7.x I can't reproduce it. it pops up within a rails application.
<fkr>
it is the content length we set for a download
<nirvdrum>
I'm suggesting it might be more complex. Are you doing something like lookup[data['content_length'].to_i] ?
<fkr>
nope
havenwood has joined #jruby
<nirvdrum>
Okay. Well, if you can make a small reproduction case, that'd be helpful. Otherwise, if you can provide backtraces, that'd be helpful.
<fkr>
nirvdrum: how can I force a backtrace?
<fkr>
because all I currently have, is this log message: 2015-07-17 14:48:06.985:WARN::/download/fcf70640-2c6b-11e5-9c7f-45ab131a9596: org.jruby.exceptions.RaiseException: (RangeError) too big for int:
<fkr>
combined with my missing clue about ruby, ain't getting me far :)
<nirvdrum>
Hmm.
<nirvdrum>
Usually when an exception occurs, you just get them.
<nirvdrum>
You could try setting JRUBY_OPTS="-Xbacktrace.style=raw"
<nirvdrum>
But you can't even see which line of code?
<fkr>
nope.
<fkr>
but I just found something. will be back in a few minutes, maybe I got it
<nirvdrum>
How do you know it's data['content_length'].to_i causing the issue then?
<fkr>
because I can comment it out and don't get the content length of my download but the server does not crash :)
<nirvdrum>
Heh. Fair enough.
<nirvdrum>
Not that it justifies the problem at all, but if your Content-Length is > 32-bits, you might want to use chunked transfer, in which case you no longer need Content-Length :-)
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] mkristian reopened issue #3133: JRuby-Complete-1.7.21 not running Rake on WIndows 7 http://git.io/vmLMq
JRubyGithub has left #jruby [#jruby]
<fkr>
nirvdrum: that is actually a very valid point!
<fkr>
I actually suspect that my problem lies in rest-client
<fkr>
(which is what we use and whose response we set the content length)
mjelen has quit [Read error: Connection reset by peer]
mjelen_ has joined #jruby
balo_ has joined #jruby
balo_ has quit [Client Quit]
samphippen has joined #jruby
samphippen has quit [Client Quit]
shellac has joined #jruby
subbu has quit [Ping timeout: 244 seconds]
mike___1234 has quit [Ping timeout: 246 seconds]
lance|afk is now known as lanceball
<fkr>
nirvdrum: thanks for your time!
<fkr>
nirvdrum: i've worked around it. not perfect, but I will look into the chunked transfers, that seems to be the proper solution.
<fkr>
nirvdrum: have a nice weekend!
fkr has left #jruby [#jruby]
enebo has joined #jruby
mike___1234 has joined #jruby
alderamin has joined #jruby
tenderlo_ is now known as tenderlove
vtunka has quit [Quit: Leaving]
kfpratt has joined #jruby
havenwood has quit [Ping timeout: 244 seconds]
havenwood has joined #jruby
camlow325 has joined #jruby
alderamin has quit []
shellac has quit [Quit: Computer has gone to sleep.]
samphippen has joined #jruby
shellac has joined #jruby
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] eregon pushed 1 new commit to master: http://git.io/vmoxP
<JRubyGithub>
jruby/master 102da2f Benoit Daloze: [Truffle] Make IO#puts write atomically the contents and the \n....
JRubyGithub has left #jruby [#jruby]
imperator has joined #jruby
<imperator>
good morning
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] eregon force-pushed truffle_rm_gil from 087113a to 83a8a7c: http://git.io/vmoxh
<JRubyGithub>
jruby/truffle_rm_gil 83a8a7c Benoit Daloze: [Truffle] No more global lock!
JRubyGithub has left #jruby [#jruby]
kfpratt has quit [Remote host closed the connection]
subbu has joined #jruby
alderamin has joined #jruby
<alderamin>
Wow, I just installed jruby 9000rc2 and ruby 2.2.2, up from jruby 1.7.14 and ruby 1.9.3. A simple test of my project code went from 700 seconds runtime to 379 seconds.
kfpratt has joined #jruby
bffff_ has joined #jruby
mjelen_ is now known as mjelen
<nirvdrum>
alderamin: Nifty. What's it do?
subbu has quit [Ping timeout: 244 seconds]
<nirvdrum>
enebo: Get anywhere with the jruby-jars failures?
<alderamin>
nirvdrum: Evaluates all possible N-combinations of objects in a set via recursion. The objects contain just two integers. Lots of small 2-dimensional arrays.
<alderamin>
nirvdrum: Looking to parallelize it now since the combinations are largely independent.
bjfish2 has joined #jruby
<chrisseaton>
alderamin: sounds like a fun benchmark - is your code open source?
<alderamin>
chrisseaton: We discussed it via email a month or so ago. I was going to let you know once I get it parallelized. I'm the DNA guy.
<chrisseaton>
aah right
<chrisseaton>
yeah sorry that's been pushed back several times - I still have to read what you're already code up
<chrisseaton>
*coded up*
<alderamin>
Don't worry about it right now.
<chrisseaton>
I had to hand my thesis in
<alderamin>
My boss is on me to get this stuff running. Even bought me a new computer to run it. He really wants this paper out.
<alderamin>
Once it gets running for the paper, I can spend time more time optimizing it.
imperator has left #jruby ["Leaving"]
<chrisseaton>
Being in a running state with tests is the best way for us to experiment with running on it on Truffle
kfpratt has quit [Remote host closed the connection]
<alderamin>
I mean it's all on git, and having multiple version of ruby installed means we can test it on all those combinations of interpreters and various commits.
<alderamin>
jruby*
subbu has joined #jruby
camlow325 has quit [Remote host closed the connection]
rcvalle has joined #jruby
benlovell has joined #jruby
subbu has quit [Ping timeout: 244 seconds]
<enebo>
nirvdrum: I have not no, but mkristian said he would look today
blaines has joined #jruby
blaines has quit [Ping timeout: 246 seconds]
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] blerins opened issue #3149: Issue requiring 'yaml' on SwingWorker on jruby 9.0.0.0.rc2 http://git.io/vmK45
JRubyGithub has left #jruby [#jruby]
<nirvdrum>
It looks like it was passing a bit this morning and then started to fail again.
<enebo>
nirvdrum: yeah but last build passed. so something intermittent
<nirvdrum>
Yeah. I just don't know if that makes it a blocker or not.
<enebo>
nirvdrum: this started failing with an inconsequential change. I don’t think this is from a commit. I think it is environmental problem
<enebo>
nirvdrum: I guess a commit earlier than first failure could have caused it too
blaines has joined #jruby
<nirvdrum>
SourceForge is still down. So that's cool.
subbu has joined #jruby
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] enebo closed issue #3149: Issue requiring 'yaml' on SwingWorker on jruby 9.0.0.0.rc2 http://git.io/vmK45
JRubyGithub has left #jruby [#jruby]
<enebo>
heh
dumdedum has quit [Quit: foo]
goyox86 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
djbkd has joined #jruby
benlovell has quit [Ping timeout: 248 seconds]
shellac has quit [Quit: Computer has gone to sleep.]
tvo has joined #jruby
tvo has quit [Changing host]
tvo has joined #jruby
mkristian has joined #jruby
<nirvdrum>
Did the psych MBC fix ever get merged?
bbrowning is now known as bbrowning_away
JRubyGithub has joined #jruby
JRubyGithub has left #jruby [#jruby]
<JRubyGithub>
[jruby] mkristian reopened issue #3149: Issue requiring 'yaml' on SwingWorker on jruby 9.0.0.0.rc2 http://git.io/vmK45
<brightball>
headius: I've got a huge REE 1.8.7 legacy site that I'm considering trying to port to jRuby. Is the benefit of moving from REE 1.8.7 to jRuby for 1.8.7 worth the level of effort?
subbu has quit [Ping timeout: 256 seconds]
<enebo>
brightball: it should work to move it but it really depends on what worth it means. If you put extra work into making it use multithreaded support in Rails it would but for an old big app that might be a big effort.
benlovell has joined #jruby
<enebo>
brightball: but who knows it may run better than REE 1.8.7 so maybe try it and see how it goes
<brightball>
enebo: that's basically the big concern. There's about 120,000 lines of code in this thing (before tests) :-)
<enebo>
heh
bbrowning_away is now known as bbrowning
mkristian has quit [Ping timeout: 252 seconds]
<enebo>
brightball: evaluating which gems you have might be a good first place. If there is something which requires a lot of effort (not a suitable replacement for a native gem) then you may have an early indicator
shellac has joined #jruby
<Antiarc>
brightball: What's your impetus for moving? Just want better speec/concurrency?
<brightball>
Antiarc: it's a legacy system that's going to "exist" for many more years but isn't being actively developed because it's replace is being actively developed. The issue is that it's still the core system that about 2 million active users depend on across iOS/Android and web (social site) and according to New Relic Ruby is very much the bottleneck accounting for 5/7 of request time on average
benlovell has quit [Ping timeout: 264 seconds]
benlovell has joined #jruby
<brightball>
we've tried upgrading rails, upgrading ruby and every time it turns into a month long process that tends to result in throwing in the towel
<Antiarc>
Well, I can say pretty confidently that jruby will be faster than 1.8.7, but so would MRI 2.x, and the latter may be a much easier upgrade
samphippen has joined #jruby
<brightball>
so at this point we're basically looking for a way to make minimal changes to the code and get it running faster for everybody else's benefit
<brightball>
I'm just finding a lot of mixed information on REE vs jRuby for 1.8.7. There doesn't seem to be much in the way of definitive out there
<Antiarc>
Well, I'd say get off of 1.8.7 either way because it's no longer supported, even for security stuff
<Antiarc>
Are you trying to upgrade straight to 2.2, or are you going 1.9 -> 2.0 -> 2.1 -> 2.2?
<brightball>
we were going to go for the latter
<Antiarc>
I'd recommend a stepped upgrade, personally
<Antiarc>
Upgrade to the next major version, get your test suite working, repeat until you're on your desired target
<Antiarc>
Have you instrumented your slow parts and figured out specifically what about them is slow? If you're actually pinpointing Ruby The Language as the slow part then you're stuck needing to upgrade, but if the system is due for retirement soon, it may make sense to just spend some time with ruby-prof and kcachegrind
camlow325 has joined #jruby
<brightball>
oh, it's cached to the hilt
<brightball>
we've separated out many parts into microservices (about 9 so far) to get their functionality out of the monolith and those are all speedy
blaines has quit [Ping timeout: 264 seconds]
<Antiarc>
I don't just mean caching, but actually understanding what in your code is slow - it might be that you have a bottleneck that's avoidable
<Antiarc>
caching puts a band-aid on slow code, but if you can fix the slow stuff under the caching, even better :)
<brightball>
any tool suggestions to help track it?
<Antiarc>
newrelic to find slow actions, then ruby-prof and kcachegrind to find out what's going on in that action
<brightball>
I jumped into Ruby when 1.9.3 was firmly established so all of the 1.8.7 upgrade issues aren't something I had to deal with until now
subbu has joined #jruby
<brightball>
and especially not in this magnitude
<brightball>
it's been a...learning experience
<Antiarc>
Fortunately the jump from 1.8 to 1.9 in a web app is a pretty small developer surface, aside from the string encoding changes
<Antiarc>
Unicode handling definitely gave me the most grief when we upgraded our stuff
mkristian has joined #jruby
<brightball>
yea, I experience that on the last upgrade attempt
<Antiarc>
Unfortunately you'll have to deal with that eventually no matter whether it's jruby or MRI
<alderamin>
Is there a good channel for concurrent programming practices? I have some more hypothetical questions that don't have anything to do with ruby.
subbu has quit [Ping timeout: 250 seconds]
<Antiarc>
most channels on freenode tend to focus on a specific technology, I think
<Antiarc>
I'd say ask away, most concurrency questions will have some grounding in implementation :)
pietr0 has quit [Quit: pietr0]
djbkd has quit [Remote host closed the connection]
lanceball is now known as lance|afk
mike___1234 has quit [Ping timeout: 255 seconds]
<alderamin>
Ok, so I have a set to perform combinations over. Each execution of my program will have a specific set of parameters to work under, including the original set, number of objects to combine, and some other parameters. Some parameters won't change, an invariant condition of my program. Do I need to worry about thread safety for these invariant parameters?
mkristian has quit [Ping timeout: 240 seconds]
<Antiarc>
You typically don't need to worry about synchronized access to values that don't change
mike___1234 has joined #jruby
<alderamin>
I suppose I could always be safe by putting write locks on them with the assumption that there will never be one after the initial value is set.
<Antiarc>
that said, if you're taking a mutex or something to get the values of your variant parameters, you may as well get the values of your invariants and cache them in local variables anyhow. Generally, taking a mutex is slow, copying variable values is fast
<Antiarc>
If you're working in a language that supports then, you could make them const values
<alderamin>
Ah I see. I'm working in pure ruby for now.
<Antiarc>
Okay, so nothing truly unmodifiable then :)
<Antiarc>
you can call .freeze if it's an array or something, but that won't protect it quite the same way const/final would
<nirvdrum>
brightball, Antiarc: When I did REE -> 1.9.3, the biggest issues I encountered were: 1) String encoding (and the changes made to the String API (e.g., each vs each_char vs each_byte); 2) changes to case syntax; and 3) massive performance degradation.
<Antiarc>
Really, 1.9 was slower for you? That's surprising
<Antiarc>
IIRC we picked up about a 25% performance boost from the 1.9 upgrade
<nirvdrum>
Mogotest was about 2.5x slower across the board.
<Antiarc>
Yikes.
<Antiarc>
Well in any case, the 1.9 -> 2.0 upgrade is trivial :)
<nirvdrum>
Anything String heavy was substantially slower.
<Antiarc>
2.0 -> 2.1 is a bit trickier
<Antiarc>
Ah, that's true
<nirvdrum>
Startup time was abysmal without the falcon patches.
<Antiarc>
Were you running with GC tuning?
<nirvdrum>
Yeah.
<Antiarc>
interesting.
<nirvdrum>
Well, 1.9.3 had the quadratic require time.
shellac has quit [Quit: Computer has gone to sleep.]
<Antiarc>
Oh, that's right, I'd forgotten about that
<Antiarc>
Haha
<nirvdrum>
And quadratic here meant lstats, so it was whacking the crap out of the filesystem.
<nirvdrum>
And byte-oriented String operations were just a lot faster than the encoding-aware stuff.
<brightball>
nirvdrum: did you go straight to 1.9.3 or 1.9.2 first?
<nirvdrum>
Basically, whatever Yehuda's microbenchmarks were didn't match reality at all for me.
donV has joined #jruby
<nirvdrum>
I think I went to 1.9.3.
<nirvdrum>
And from there I quickly went to JRuby.
enebo has quit [Quit: enebo]
<nirvdrum>
But, the other part with JRuby is if you can't multithread your app, you need to have the resources to run a bunch of Java processes. You don't want to keep forking the JVM.
<brightball>
yea, agree there
<brightball>
as big as this system is, my assumption is that the threading alone would be huge
<brightball>
because it is heavy on the forks with such a huge code base
<nirvdrum>
My feeling is if you're going to do MRI, just skip 1.9.x :-) 2.0 isn't that much different and you'll see a noticeable performance bump. And 1.9.3 is also unsupported *shrug*
<brightball>
wonder how many people have attempted 1.8.7 to 2.0 jumps successfully
<Antiarc>
It's probably not too bad, aforementioned problems aside
<Antiarc>
If you have a good test suite it's likely not an awful transition
mkristian has joined #jruby
<nirvdrum>
Oh, crap. I forgot about that.
<nirvdrum>
That's why I hated that upgrade so much.
<Antiarc>
I personally try to do smallish incremental upgrades, with commits for each
<nirvdrum>
Actually, I take that back, that was my Rails 2.2 -> 3.0 upgrade that required me to also upgrade rspec 2 -> 3.
<Antiarc>
so that my change surface is as small as is reasonable
<nirvdrum>
That's a good approach. Going to 1.9.3 is a reasonable step. I'm just suggesting if you go that route, you might find performance to degrade and question the whole process :-)
djbkd has joined #jruby
<brightball>
okay, so one last question for the people with experience
<brightball>
when upgrading a rails 2.3 and ree 1.8.7 - what should be upgrade first
<brightball>
rails to 3.0 or ruby to 1.9.3?
<Antiarc>
Is 3 even supported on 1.8?
<nirvdrum>
It is.
<Antiarc>
Looks like it is
<nirvdrum>
And the code generators generated different code based on whether it detected running in 1.8 or 1.9.
<Antiarc>
I'd probably go 1.9 -> 2.0, then rails 2 -> 3
<Antiarc>
All that said, you said this project has its replacement being developed
<Antiarc>
I wouldn't sink a TON of time into it
<Antiarc>
It may be cheaper to just buy some more hardware and throw that into the mix
<Antiarc>
(developer time is costly!)
<nirvdrum>
I'd probably stay on Rails 2.3 if you can, unless you really need something newer. That upgrade is going to be far more annoying than the Ruby upgrade.
<brightball>
it's complicated because this project is still being used by 2 million people and will be around for at least 7 or 8 more years
<nirvdrum>
2.3 was simple enough that you could patch it on your own, too.
<brightball>
the "replacement" is a new product that we'll eventually work to migrate people over to
<Antiarc>
Oh, so this one is going to be in place for a while then
<brightball>
yea, basically just looking at it from an infrastructure cost vs developer time perspective
<Antiarc>
I agree with nirvdrum's assessment - the rails upgrade will be far worse than the ruby upgrade
<brightball>
got it
<brightball>
thanks for the help. Gotta hop off and meet somebody for lunch
<nirvdrum>
Essentially, to go from Rails 2.3 -> 4.2, you may as well consider any other framework your heart desires. So much is going to change that it's a borderline rewrite.
<nirvdrum>
Your entire dependency graph is going to have to upgrade along with it.
brightball has quit [Quit: Leaving...]
<Antiarc>
If I was in charge of it, I would a) get all the low-hanging fruit from the existing app with ruby-prof/kcachegrind, b) add more hardware, then c) upgrade to Ruby 2.0+ if I didn't have the perf I wanted after that (or JRuby if concurrency is a concern)
blaines has joined #jruby
<donV>
Hi all!
<donV>
kares: Hi! You on? Seen my progress?
<kares>
donV: yeah I did not get a PR that was open for 10 mins :)
<kares>
good work
<kares>
have to go now, will do deeper review as I merge (hopefully next week) and try to do similar changes at least for H2/HSQLDB
<donV>
kares: Many changes I did were on general jdbc.
<kares>
yy
<kares>
noticed
chamila has joined #jruby
rsim1 has quit [Quit: Leaving.]
lance|afk is now known as lanceball
pietr0 has joined #jruby
yfeldblum has joined #jruby
temporal_ has joined #jruby
temporalfox has quit [Ping timeout: 246 seconds]
mjelen_ has joined #jruby
mjelen has quit [Ping timeout: 260 seconds]
bbrowning is now known as bbrowning_away
mjelen_ has quit [Remote host closed the connection]
mjelen has joined #jruby
mjelen has quit [Read error: Connection reset by peer]
<alderamin>
I'm running 75 Promises on a thread pool with :max_threads => 8. Watching Task Manager, java never gets about 15%, and that's presumably only due to GC. What do I need to do in order to get them to run concurrently in real-time?
<alderamin>
They're all accessing invariant variables that are not thread-safe objects.
<alderamin>
Could that be it?
mjelen has quit [Ping timeout: 248 seconds]
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] mkristian closed pull request #3127: Added uri to path when using file protocol to chdir (jruby-1_7...jruby-1_7) http://git.io/vqS7F
JRubyGithub has left #jruby [#jruby]
xkickflip has quit [Quit: xkickflip]
<mkristian>
nirvdrum, I need to include javadocs along side the sources otherwise I just produce to many knots in the build. currently the 9.0.0.0 release got stuck with missing javadocs
<mkristian>
I hope you can live with this for -Psnapshots and some IDEs do appreciate this as well
<nirvdrum>
mkristian: No worries. I don't think we need them, so I was just going to save the time on building. But it's an async process, so not a big deal.
<nirvdrum>
The intern seemed happy :-)
<nirvdrum>
alderamin: Do the Promises finish their work quickly? Or do they do a lot of I/O?
<nirvdrum>
Unless the process is CPU bound, you won't see utilization climb.
<alderamin>
It is heavily bound by CPU.
<alderamin>
The only I/O in the program is populating those invariant variables before the promises are created, and the only output is to the screen after they've finished.
<Antiarc>
alderamin: How many cores in the machine?
<alderamin>
Antiarc: i7, four physical cores but 2 threads/core.
<Antiarc>
alderamin: How long does this run, approximately?
<alderamin>
Antiarc: It should run for 30 seconds in a single thread.
<Antiarc>
alderamin: one thing you might try is attaching to your JVM process with jvisualvm and using the CPU profiling to see how many threads there are and what they're doing
<Antiarc>
How are you queueing jobs in the pool?
<alderamin>
Antiarc: 75, but also trying 150 results in the same.
<alderamin>
Antiarc: Oh, sorry. I'm creating an array of the Promises with fulfill(), then iterating over that array and .execute'ing each of them.
subbu has quit [Ping timeout: 264 seconds]
<alderamin>
I then iterate over the array to collect .value, which I imagined would block as needed.
<Antiarc>
(I've only used Futures in this context; looking for doc on Promises)
<Antiarc>
How are you setting up your promises? It's possible that you're doing your worn on your main thread somehow
<Antiarc>
If it's something like Promise.fulfill(some_expensive_method) that's not going to work, since it has to run some_expensive_method before its result can be passed to Promise.fulfill
<Antiarc>
You would want something like Promise.new { some_expensive_method } instead
kfpratt has joined #jruby
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] mkristian opened issue #3150: File.expand_path( 'asd', Pathname.new('uri:classloader:/') produces corrupted result http://git.io/vm695
JRubyGithub has left #jruby [#jruby]
benlovell has quit [Ping timeout: 252 seconds]
bbrowning_away is now known as bbrowning
mike___1234 has quit [Ping timeout: 256 seconds]
<alderamin>
Antiarc: Awesome, I was looking into Futures and wondered if that could be the case.
<Antiarc>
alderamin: Promises are almost certainly implemented with Futures (or something like it) under the hood
<Antiarc>
I suspect you're just doing your expensive work on your main thread rather than via the executor
subbu has joined #jruby
benlovell has joined #jruby
mjelen_ has quit [Remote host closed the connection]
benlovell has quit [Ping timeout: 264 seconds]
<alderamin>
Antiarc: Still not quite getting, but I think i'm a lot closer now. Thanks for your help.
<alderamin>
Everyone have a good weekend.
<Antiarc>
alderamin: sounds like you're heading out, but if you are still having trouble, gist your code and I'll see if I can help you nail it down :)
<alderamin>
Antiarc: Will do. Probably won't look at this again until Sunday. Thanks again for all of your help.
<alderamin>
The ~2x speed improvement from just using current stable releases is exciting enough to make today a net positive. Getting 8 threads running will be amazing.
<alderamin>
As my code sits, I'll need 75 days running 24/7 to get all the results we need for this paper. If I get a 6-8x improvement with running it over 8 physical threads, we'll be down to less than a week, including the interpreter boosts :)
<Antiarc>
alderamin: IMO, rent out an Amazon cc3.8x, 32 cores and 60 GB of RAM
<Antiarc>
You can get spot instances for about $0.25/hour
JRubyGithub has joined #jruby
<JRubyGithub>
[jruby] mkristian pushed 1 new commit to master: http://git.io/vm6jG
<JRubyGithub>
jruby/master 697d33e Christian Meier: [build] get the javadoc for jruby-complete and -Psnapshots in place again
JRubyGithub has left #jruby [#jruby]
<alderamin>
Antiarc: Told my boss that, he authorized me a new i7 instead. I'm not complaining.
<Antiarc>
hah, fair enough
mike___1234 has joined #jruby
<Antiarc>
We run lots of really CPU-intensive stuff in JRuby on those and they work like a charm :)
<alderamin>
Antiarc: To be fair, I'll be using it to get into hadoop for my masters thesis, hopefully.
<Antiarc>
Something else to consider, if you have lots of numeric work, consider writing a Java class and calling it from Ruby rather than doing it in Ruby
<Antiarc>
We see an ~8x performance improvement when we did that
<mberg>
The ability to refactor chokepoints in Java is one of the best arguments for JRuby.
<Antiarc>
That, and jvisualvm makes finding those chokepoints trivial
<alderamin>
Antiarc: little arithmetic in this code, but our data is represented in integer values, and we have to count billions or more of observed values over fairly short arrays.
yfeldblum has quit [Remote host closed the connection]
<mberg>
And all the wonderous concurrency and threading built into the JVM.
<alderamin>
I should say, all the arithmetic is in counting numbers.
goyox86 has joined #jruby
<Antiarc>
Sounds like lots of allocations - reimplementing some of it in JAva may produce substantial improvements. But, measure first :)
<mberg>
But yeah, even without writing much real java, just using existing packages, has often yielded an 8-12x increase for me.
<alderamin>
mberg: That's why I'm using concurrent-ruby :)
xkickflip has joined #jruby
<mberg>
JavaMail, HttpComponents, UnboundID are all great friends of mine.
<Antiarc>
HttpComponents is great - Manticore is a much better HTTP client than all the rest of its Ruby siblings as a result, IMO :)
<Antiarc>
It's just a giant pain to work with. Which is why JRuby is so nice - being able to wrap Java libs in shiny Ruby wrapping is delightful
alderamin has quit []
subbu has quit [Ping timeout: 244 seconds]
<mberg>
Yeah, I've been meaning to play more with Manticore instead of directly in HttpComponents, but it seems like I'm constantly having to do Weird Things. Which is another reason for using HttpComponents in the first place :D
<Antiarc>
fwiw, if you find weird things you can't do with it, let me know or file a ticket - I want to make it flexible enough to handle those!
<mberg>
You may even account for them now. :D
<Antiarc>
The goal is to wrap up the 95% case in a nice concise interface, and then expose the right hooks so the people with really unique needs can get in there easily
<Antiarc>
What are some of them?
<mberg>
One of them was doing a multipart/form post even though the parameters were just simple key/value pairs and would normally be done as query parameters.
<mberg>
Another was an API where they insisted I provide unsolicited auth credentials on the first connect
enebo has joined #jruby
kfpratt has quit [Remote host closed the connection]
<mberg>
IIRC that one I had to create an HttpClientContext, set a CredentialsProvider, create a pre-populated AuthCache, and some other jiggery-pokery to get it to work.
<Antiarc>
requests take an :entity parameter which can be an instance of any Entity, or a :body which creates a StringEntity, so that should cover the first case
<Antiarc>
I don't think we prepopulate an AuthCache but you can set credentials on a request to be sent unsolicited; not sure if that would cover that case or not
<mberg>
I'm vaguely remembering bringing that use case up a while back when you were talking about Manticore. Would have been over a year ago.
<mberg>
For the auth stuff, it seemed to be that the Apache folks deliberately omitted offering credentials on the initial connect on purpose, citing it as bad practice (which of course it is).
<mberg>
Found some people who resorted to just creating the headers manually and setting them on the request.
<Antiarc>
Hm, I thought that's what the AuthScope was for - you specify who the creds can be offered to, and then it can offer them
<Antiarc>
I'll have to look into that more specifically
camlow325 has quit [Remote host closed the connection]
camlow325 has joined #jruby
<Antiarc>
it wouldn't be hard to prepop an authcache in response to a flag
camlow325 has quit [Remote host closed the connection]
<mberg>
It's entirely possible they've since accounted for it. But when I wrestled with it, pretty sure that setting it in an AuthScope would still have it wait for a 401 before offering credentials.
<Antiarc>
interestiong
<Antiarc>
I'll have to set up a unit test to see what the behavior is
<mberg>
I'm going to be playing with it again soon. The need has arisen for a sane client library for that API.
<mberg>
They have a python library but it sucks. And it's python.
<Antiarc>
I'll set up tests if I remember to. If you run into problems though, do let me know!
<mberg>
Ah, here's the doc mention: "HttpClient does not support preemptive authentication out of the box, because if misused or used incorrectly the preemptive authentication can lead to significant security issues, such as sending user credentials in clear text to an unauthorized third party. Therefore, users are expected to evaluate potential benefits of preemptive authentication versus security risks in the context of their specific application environment."
<mberg>
"Nonetheless one can configure HttpClient to authenticate preemptively by prepopulating the authentication data cache."
<Antiarc>
Got it. Makes sense. I'll see about testing it and adding a flag to prepopulate the auth cache.
<Antiarc>
at any rate, that illustrates why I love HttpComponents
<Antiarc>
It's so comprehensive that they've considered that use case, designed for their response, and offer an alternative
goyox86 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<mberg>
Yep. It makes the hard possible. It also makes the easy hard, but that can be forgiven. Especially with Fluent offered for the simple case.
<Antiarc>
Java seems to have a habit of making the easy hard. But that's why JRuby is so nice - you can take that hard stuff, make it easy, and get all the benefits without the downsides :)