huoxito has quit [Remote host closed the connection]
huoxito has joined #rubygems
huoxito has quit [Ping timeout: 240 seconds]
arohner has quit [Read error: Connection reset by peer]
arohner has joined #rubygems
flaccid has joined #rubygems
<flaccid>
i'm on osx mountain lion. if i use irb and require 'foo' it works, but for some reason if i do it in a script with /usr/bin/ruby or /usr/bin/env ruby it says no such file to load
<indirect>
gem install --verbose _might_ be helpful?
<indirect>
but we know what the error is
<indirect>
and where it's coming from
<indirect>
unfortunately the output you pasted is just the result of a bug in bundler where I incorrectly call _all_ SSL errors "certificate validation failures"
<indirect>
that's not actually what's happening
<indirect>
you're probably actually getting the "wrong version" ssl error that I opened a ticket about
tekin has quit [Remote host closed the connection]
einarj has joined #rubygems
jaimef has quit [Excess Flood]
lsegal has joined #rubygems
jaimef has joined #rubygems
cowboyd has joined #rubygems
jkline has quit [Quit: jkline]
dvu_ has quit [Remote host closed the connection]
huoxito has joined #rubygems
havenwood has quit [Remote host closed the connection]
dvu has joined #rubygems
eugenejen has left #rubygems [#rubygems]
bradland has joined #rubygems
bradland has quit [Quit: bradland]
bradland has joined #rubygems
havenwood has joined #rubygems
bradland has quit [Client Quit]
fromonesrc_ has quit [Ping timeout: 260 seconds]
tcopeland has quit [Quit: Leaving.]
redmenace has joined #rubygems
arthurnn has quit [Remote host closed the connection]
arthurnn has joined #rubygems
arthurnn has quit [Read error: Connection reset by peer]
arthurnn_ has joined #rubygems
redmenace has quit [Client Quit]
redmenace has joined #rubygems
stevenharman has quit [Quit: Leaving...]
tjbiddle has joined #rubygems
arthurnn_ has quit [Remote host closed the connection]
almostwhitehat has quit []
stevenharman has joined #rubygems
jnimety_ has joined #rubygems
jnimety has quit [Read error: Operation timed out]
jnimety_ has quit [Quit: Computer has gone to sleep.]
cowboyd has quit [Remote host closed the connection]
redmenace has quit [Ping timeout: 260 seconds]
redmenace has joined #rubygems
workmad3 has quit [Ping timeout: 240 seconds]
einarj has quit [Remote host closed the connection]
turnip_jl has joined #rubygems
<turnip_jl>
evan: hello, I'm @jonleighton, I presume you're @evanphx ?
<evan>
just evan!
<turnip_jl>
:)
<evan>
the benefit of 15+ years on freenode
<evan>
we're going to try a few things.
<evan>
this is much easier than coordinating over twitter.
<indirect>
ha
<indirect>
good plan
<turnip_jl>
hehe, yeah twitter's not the best for interactive debugging
<evan>
ok, I want you to change to hit rubygems.org on port 444
<evan>
go ahead and start
<evan>
wait 5s between each request
<evan>
was that a failure?
<evan>
that last one
<turnip_jl>
nope
<evan>
one good, one bad
<evan>
ok
<evan>
one sec, let me change something
<turnip_jl>
ok
<josh-k>
hey evan
<evan>
hey
<josh-k>
ohai turnip_jl
<turnip_jl>
josh-k: wassup darling
<josh-k>
evan: how you going buddy?
<josh-k>
turnip_jl: hey honey
<josh-k>
it's a real party in here!
<evan>
i'm going to use tshark instead of tcpdump
<evan>
one sec.
<josh-k>
is indirect here too!
<josh-k>
wow, this is almost better than real life!
<evan>
turnip_jl: ok, start again.
<turnip_jl>
evan: do you have any idea why https://gist.github.com/jonleighton/7099162 shows the ssl error against s3, not rubygems? i.e. are you sure rubygems.org is the problem?
<turnip_jl>
starting
tcopeland has joined #rubygems
<evan>
ok, stop
<evan>
sorry
<evan>
i'm trying to get this configured so I can see more details
<turnip_jl>
evan: that one failed
<evan>
one sec.
<evan>
ok
<evan>
ok, start again
<evan>
i'm going to capture the traffic this time
<evan>
so I can analyze it offline
<evan>
let me know once you've got a few success and a few failures.
<evan>
got them?
<turnip_jl>
no failures yet
<evan>
ok
<turnip_jl>
the failures aren't that common with s_client (compared to how frequently I get them when bundling)
<evan>
ok
<evan>
the fact you see errors on both rubygems and S3
<evan>
feels like it's something on your end
huoxito has quit [Remote host closed the connection]
<evan>
but so many people are seeing this
<evan>
you'd think it would be on the server side
<turnip_jl>
yeah it's really strange
<turnip_jl>
evan: ok that was a failure
<turnip_jl>
you want me to continue?
<evan>
nah, thats fine
<evan>
give me a few minutes to decode it.
<drbrain>
evan: I heard rumors of a bad cert in OpenSSL can cause the "bad version number" error
<turnip_jl>
fwiw the failure seem to hang a bit after the "CONNECTED(00000003)" line
<indirect>
evan: I'm also pretty sure it's server-side due to headius being able to reproduce at will from JRuby on his development machine
<evan>
drbrain: but what is a bad cert?
<evan>
if it were that
<evan>
you'd think it would be every time
<drbrain>
evan: no idea
<drbrain>
thus, "rumors"
<evan>
indirect: against rubygems.org or s3 though?
<evan>
and did he have RG upgraded to deal with the S3 cert change
<evan>
there are 2 SSL issues going on here
<evan>
it's important we separate them out
<evan>
they're not the same.
<indirect>
evan: we should ask headius those questions on his ticket; it's possible his problem is the s3 cert one
<evan>
I can't keep track of who said what on which ticket anymore
<evan>
:( :(
<indirect>
also let me know if it would be helpful for me to run tests from travis
<drbrain>
(extra confusing for this issue is that people report "SSL failure" but not the specific message, or comment on the wrong specific message)
<indirect>
I have direct access to a spare travis VM for this
<evan>
drbrain: nor which host they got it from.
<turnip_jl>
drbrain: unfortunately bundler just reports "Could not verify the SSL certificate..." so it's not very specific about the problem
<evan>
cloudshark isn't good enough, wireshark for OS X here I come!
<evan>
oh good, I have to install X
huoxito has joined #rubygems
<turnip_jl>
evan: I kind need to go sleep, it's midnight here sorry :/
<evan>
thats fine
<evan>
i've got data to try and analyze
<evan>
nite!
<evan>
thanks for the help
<indirect>
turnip_jl: I have fixed that bug in the latest bundler, but it's not going to get much more specific
<turnip_jl>
cool thanks. happy to help again another day. my email is j@jonathanleighton.com if you need it.
<indirect>
the ticket in question includes the underlying ruby openssl exception
<turnip_jl>
indirect: nice one
turnip_jl has left #rubygems [#rubygems]
<evan>
ok, nginx seems to send a RST response to the SSL Client Hello
<evan>
that hello looks exactly like a hello that it happily handled.
<evan>
blarg.
<evan>
actually
<evan>
there is one thing.
<evan>
the hello that failed has 10 seconds different between the receive time and the timestamp inside the SSL session
<indirect>
clock skew?
<evan>
the one that did work has a sub-second difference.
<indirect>
I'm pretty sure that big of a time difference will make it not work
<evan>
really?
<evan>
I didn't think SSL checked that time field
<evan>
it's just part of the random vector used to create the session
<indirect>
oh
<indirect>
nevermind... looks like it's on the order of hours
<indirect>
I knew I had seen references to SSL connections failing due to clock skew in the past, but that makes me think that that is not our problem now
<evan>
yeah
<indirect>
so maybe the 10s is indicative of a network issue?
<evan>
thats a good call.
<evan>
oooh.
<evan>
I have a theory.
<evan>
the TCP handshake took 10 seconds
<evan>
on this failed request.
<evan>
:08.68 => SYN
<evan>
:08.68 => SYN+ACK
<evan>
:08.85 => ACK
<evan>
at this point, nginx is woken up and given the socket I believe (unless the option to wait until first data is used, dunno if nginx does)
<evan>
:13.85 => FIN! (server to client)
<evan>
:14.02 => ACK to FIN (client to server)
<evan>
:18.89 => first packet with data
<evan>
wow, what a fucked up tcp session.
<evan>
nginx has a linger_timeout directive that is set to 5s by default
<evan>
that must be what closed the socket, thus the FIN
<evan>
it's exactly 5s between ACK and FIN
einarj has joined #rubygems
einarj has quit [Ping timeout: 245 seconds]
<evan>
AAARG
<evan>
found it.
<evan>
client_header_timeout is set to 5
<evan>
you know, this is probably my fault.
<evan>
i'll bet I changed it by hand on the old LB.
<evan>
maybe.
<evan>
who knows.
<evan>
anywho
<evan>
no one is listening anyway.
<drbrain>
I AM‼‼
<evan>
thanks bud
<drbrain>
I knew you would fix it once you had gathered enough round tuits‼