drumsrgr8forn8 has joined #rubygems
DanKnox is now known as DanKnox_away
drumsrgr8forn8 has quit [Ping timeout: 272 seconds]
havenwood has quit [Remote host closed the connection]
dwknoxy has joined #rubygems
ZachBeta has joined #rubygems
havenwood has joined #rubygems
sindork has quit [Quit: No Ping reply in 180 seconds.]
sindork has joined #rubygems
dvu has quit [Remote host closed the connection]
dvu has joined #rubygems
dvu has quit [Ping timeout: 245 seconds]
ZachBeta has quit [Quit: Computer has gone to sleep.]
redmenace has quit [Ping timeout: 240 seconds]
jkline has quit [Quit: jkline]
TheMoonMaster_ has quit [Excess Flood]
TheMoonMaster has joined #rubygems
<
shaiguit1r>
just set up the web for that
<
shaiguit1r>
lemme know if you have any feedback on that
shaiguit1r is now known as shaiguitar
<
shaiguitar>
evan: dwradcliffe samkottler ^ fyi
dvu has joined #rubygems
lsegal has joined #rubygems
imperator has quit [Quit: Valete!]
josh-k_ has quit [Remote host closed the connection]
josh-k has joined #rubygems
josh-k has quit [Ping timeout: 272 seconds]
TheMoonMaster has quit [Excess Flood]
TheMoonMaster has joined #rubygems
arohner has quit [Remote host closed the connection]
arohner has joined #rubygems
arohner has quit [Ping timeout: 246 seconds]
havenwood has quit [Remote host closed the connection]
redmenace has joined #rubygems
havenwood has joined #rubygems
knappe has quit [Quit: knappe]
knappe has joined #rubygems
knappe has quit [Quit: knappe]
havenwood has quit [Remote host closed the connection]
ddv has quit [Ping timeout: 252 seconds]
ddv has joined #rubygems
arohner has joined #rubygems
redmenace has quit [Ping timeout: 256 seconds]
arohner has quit [Remote host closed the connection]
lsegal has quit [Quit: Quit: Quit: Quit: Stack Overflow.]
arohner has joined #rubygems
arohner has quit [Remote host closed the connection]
lsegal has joined #rubygems
arohner has joined #rubygems
jonahR has joined #rubygems
jonahR has quit [Client Quit]
redmenace has joined #rubygems
arohner has quit [Remote host closed the connection]
arohner has joined #rubygems
arohner has quit [Ping timeout: 272 seconds]
arohner has joined #rubygems
redmenace has quit [Ping timeout: 248 seconds]
csaunders has quit [Ping timeout: 272 seconds]
danp has quit [Ping timeout: 245 seconds]
niska` has quit [Ping timeout: 245 seconds]
Emily has quit [Ping timeout: 264 seconds]
Boohbah has quit [Ping timeout: 240 seconds]
mpapis has quit [Ping timeout: 252 seconds]
Boohbah has joined #rubygems
niska has joined #rubygems
csaunders has joined #rubygems
danp has joined #rubygems
Emily has joined #rubygems
mpapis has joined #rubygems
arohner has quit [Remote host closed the connection]
redmenace has joined #rubygems
lsegal has quit [Quit: Quit: Quit: Quit: Stack Overflow.]
redmenace has quit [Ping timeout: 256 seconds]
arohner has joined #rubygems
indirect_ has quit [Quit: Oh, bother.]
Vendethi_ has joined #rubygems
arohner has quit [Remote host closed the connection]
einarj has joined #rubygems
Locke23rus has joined #rubygems
indirect_ has joined #rubygems
Vendethi_ has quit [Remote host closed the connection]
Vendethi_ has joined #rubygems
adambeynon has joined #rubygems
Vendethi_ has quit [Ping timeout: 264 seconds]
Vendethi_ has joined #rubygems
workmad3 has joined #rubygems
workmad3 has quit [Read error: Operation timed out]
josh-k has joined #rubygems
josh-k has quit [Remote host closed the connection]
josh-k has joined #rubygems
josh-k has quit [Read error: Operation timed out]
josh-k has joined #rubygems
Vendethi_ has quit [Remote host closed the connection]
niska has quit [Read error: Connection reset by peer]
niska has joined #rubygems
Glass_saga has quit [Remote host closed the connection]
Glass_saga has joined #rubygems
Vendethi_ has joined #rubygems
josh-k has quit [Read error: Connection reset by peer]
josh-k has joined #rubygems
TheMoonMaster has quit [Ping timeout: 246 seconds]
TheMoonMaster has joined #rubygems
cowboyd has joined #rubygems
huoxito has joined #rubygems
knappe has joined #rubygems
almostwhitehat has joined #rubygems
Vendethi_ has quit [Ping timeout: 252 seconds]
huoxito has quit [Remote host closed the connection]
huoxito has joined #rubygems
huoxito has quit [Ping timeout: 256 seconds]
ZachBeta has joined #rubygems
josh-k has quit [Remote host closed the connection]
josh-k has joined #rubygems
josh-k has quit [Ping timeout: 272 seconds]
jonahR has joined #rubygems
Vendethi_ has joined #rubygems
lsegal has joined #rubygems
huoxito has joined #rubygems
josh-k has joined #rubygems
huoxito has quit [Remote host closed the connection]
josh-k has quit [Remote host closed the connection]
josh-k has joined #rubygems
josh-k has quit [Ping timeout: 245 seconds]
tbuehlmann has joined #rubygems
dwknoxy has joined #rubygems
Vendethi_ has quit [Remote host closed the connection]
TheMoonMaster has quit [Excess Flood]
TheMoonMaster has joined #rubygems
josh-k has joined #rubygems
drumsrgr8forn8 has joined #rubygems
arohner has joined #rubygems
einarj has quit [Remote host closed the connection]
drumsrgr8forn8 has quit [Ping timeout: 272 seconds]
imperator has joined #rubygems
TheMoonMaster has quit [Excess Flood]
josh-k has quit [Remote host closed the connection]
TheMoonMaster has joined #rubygems
josh-k has joined #rubygems
huoxito has joined #rubygems
josh-k has quit [Read error: No route to host]
josh-k has joined #rubygems
huoxito_ has joined #rubygems
huoxito has quit [Read error: Connection reset by peer]
davispuh has joined #rubygems
josh-k has quit [Remote host closed the connection]
lsegal has quit [Read error: Connection reset by peer]
lsegal has joined #rubygems
DanKnox_away is now known as DanKnox
<
dwradcliffe>
did someone run a deploy or something?
<
davispuh>
what? :|
<
dwradcliffe>
davispuh: sorry, that wasn't a response to your question
arohner has quit [Remote host closed the connection]
x1337807x has joined #rubygems
josh-k has joined #rubygems
<
dwradcliffe>
shaiguitar: ping
einarj has joined #rubygems
drumsrgr8forn8 has joined #rubygems
brycek has joined #rubygems
hmsimha has joined #rubygems
<
qrush>
looks like there was some downtime?
<
hmsimha>
does anyone have an idea why bundle update would never complete on 'installing ffi (1.9.3)' ?
huoxito_ has quit [Remote host closed the connection]
<
dwradcliffe>
qrush: yes
einarj has quit [Ping timeout: 246 seconds]
drumsrgr8forn8 has quit [Ping timeout: 272 seconds]
huoxito has joined #rubygems
Locke23rus has quit [Remote host closed the connection]
arohner has joined #rubygems
jkline has joined #rubygems
ZachBeta has quit [Ping timeout: 256 seconds]
einarj has joined #rubygems
ZachBeta has joined #rubygems
<
evan>
we've got an instance in us-west-2
<
evan>
that is scheduled to be decomm'd by AWS
<
dwradcliffe>
which instance?
huoxito has quit [Remote host closed the connection]
<
evan>
I didn't think we were using us-west-2
<
evan>
the AWS console is all confused about it too
<
evan>
oh, events. nm.
<
evan>
it's not confused.
huoxito has joined #rubygems
<
evan>
thats dbmaster01
<
dwradcliffe>
sweeeeet
<
dwradcliffe>
date?
x1337807x has quit [Ping timeout: 246 seconds]
ezkl has quit [Quit: QUIT!]
<
evan>
I got the notice on the 21st
<
evan>
didn't get around to that email until today sadly.
<
evan>
we could do an outage window to make it easy
<
evan>
how do we want to handle it?
huoxito has quit [Read error: No route to host]
<
dwradcliffe>
aim for saturday?
<
evan>
tomorrow would be better honestly.
huoxito has joined #rubygems
<
evan>
either that or saturday night
josh-k has quit [Remote host closed the connection]
josh-k has joined #rubygems
<
evan>
the maint window won't be very long
<
evan>
get the instance ready, go into maint, dump the db, load, change app's configure, done.
<
evan>
10 mins max i'd think.
<
dwradcliffe>
we can prep the instance today
josh-k has quit [Read error: Connection reset by peer]
josh-k has joined #rubygems
<
dwradcliffe>
I'd say window of 20, but aim for <10
huoxito has quit [Remote host closed the connection]
<
dwradcliffe>
evan: can you start a new instance?
jkline has quit [Quit: jkline]
ZachBeta has quit [Quit: Computer has gone to sleep.]
tbuehlmann has quit [Quit: Konversation terminated!]
<
evan>
ok, it's booting now.
<
evan>
i'll run the default roles on it to get the accounts setup
jonahR has quit [Quit: jonahR]
huoxito has joined #rubygems
<
evan>
so i guess the procedure used to bootstrap a node isn't documented?
<
dwradcliffe>
should be in the readme
<
dwradcliffe>
something like this knife bootstrap -d chef-solo -x $DEPLOY_USER --sudo $SERVER
<
dwradcliffe>
might not be spelled out very clearly
<
evan>
and... it doesn't work
<
evan>
"cap ec2 bootstrap"
<
dwradcliffe>
try the knife command directly
jonahR has joined #rubygems
<
evan>
ok, got bootstrapped
<
evan>
now complaining about solo.rb being missing...
<
dwradcliffe>
do you have the latest code?
<
dwradcliffe>
and latest gems
<
evan>
where the heck does solo.rb appear from?
<
evan>
it's not in the tar.gz that is made from the chef dir...
<
dwradcliffe>
solo.rb is deprecated
<
evan>
it can't be.
<
evan>
nothing runs then.
<
dwradcliffe>
knife solo cook $DEPLOY_USER@$SERVER chef/nodes/dbmaster.rubygems.org.json -i $DEPLOY_SSH_KEY -N dbmaster02
huoxito has quit [Remote host closed the connection]
<
evan>
are we using knife solo now?
<
evan>
but thats not the procedure
<
evan>
thats documented
<
evan>
i'm fine with that
huoxito has joined #rubygems
<
evan>
but we all need to be the same page here.
<
dwradcliffe>
we're halfway there :( PR 140, 138
<
evan>
we can't be half way there.
huoxito has quit [Remote host closed the connection]
<
evan>
that can't be in master
<
evan>
I see sam nuked solo.rb and added knife solo
<
evan>
we need to finish it now then.
<
evan>
what is not done?
<
dwradcliffe>
docs should be much clearer now
<
evan>
these giant solo commands suck :(
<
evan>
though i'm happy to be rid of the weird env vars.
<
dwradcliffe>
should be faster too since it's just syncing the cookbooks
<
qrush>
saw the bit about the instance shuttin' down
<
qrush>
might be a good opportunity to show others the process of booting, provisionining, etc
<
evan>
let's not do it now
<
evan>
i want to cycle all the instance soon
<
evan>
but i don't want to go through that all now.
<
evan>
dwradcliffe: rsync: mkdir "/var/cache/chef/cookbooks-1" failed
<
evan>
when doing solo cook
<
evan>
is that something bootstrap is supposed to create?
<
dwradcliffe>
yeah I think so
<
evan>
i'll make it by hand and see if it gets further.
<
dwradcliffe>
wiki updated
<
evan>
looks like it's running now
<
evan>
your account should be active now
<
evan>
chef is installing packages
<
evan>
it's ec2-54-212-68-175.us-west-2.compute.amazonaws.com
almostwhitehat has quit [*.net *.split]
Glass_saga has quit [*.net *.split]
shaiguitar has quit [*.net *.split]
tmacedo has quit [*.net *.split]
khaase has quit [*.net *.split]
ZachBeta has joined #rubygems
<
dwradcliffe>
I see postgres running
<
evan>
I bootstrapped it as a dbmaster role
<
evan>
looks like PG errored out though.
<
evan>
i'll let you set it up now
<
evan>
chef didn't finish here
<
evan>
STDOUT: * Restarting PostgreSQL 9.2 database server
<
evan>
* Error: pid file is invalid, please manually kill the stale server process.
<
dwradcliffe>
I've seen that before
workmad3 has joined #rubygems
havenwood has joined #rubygems
hmsimha has quit [Quit: Page closed]
ZachBeta has quit [Quit: Computer has gone to sleep.]
jalessio has joined #rubygems
ezkl has joined #rubygems
huoxito has joined #rubygems
ZachBeta has joined #rubygems
Emily has joined #rubygems
almostwhitehat has joined #rubygems
Glass_saga has joined #rubygems
tmacedo has joined #rubygems
shaiguitar has joined #rubygems
khaase has joined #rubygems
<
dwradcliffe>
samkottler: when/how/where did you setup the pg data mount info on dbmaster01?
<
samkottler>
dwradcliffe: the first time?
<
dwradcliffe>
pg config is ok now, but the actual mount isn't there
<
samkottler>
I think just an entry in /etc/fstab
<
samkottler>
dwradcliffe: you have to manually attach the vol and then mkfs.xfs onto it
huoxito has quit [Remote host closed the connection]
huoxito has joined #rubygems
<
dwradcliffe>
samkottler: can you write up docs on that sometime today?
<
samkottler>
dwradcliffe: I probably won't get to it today
<
samkottler>
here's how to do it though if you want to write a wiki
<
samkottler>
1. create a biggish volume (250GB or something)
<
samkottler>
2. attach it to the running instance at /dev/sdg (or the first available)
<
samkottler>
3. sudo apt-get xfsprogs
<
samkottler>
4. sudo mkfs.xfs /dev/sdg
<
samkottler>
5. add /dev/xvdg to /etc/fstab with /var/lib/pg_data as the target
<
samkottler>
dwradcliffe: note that the devices are difference in 4 and 5
<
samkottler>
that's because of a kernel change
<
dwradcliffe>
xvdg or something
<
dwradcliffe>
right?
<
dwradcliffe>
oh yeah, you said that
ZachBeta has quit [Quit: Computer has gone to sleep.]
huoxito has quit [Ping timeout: 265 seconds]
<
dwradcliffe>
we should move that into chef - aws cookbook will do that I think
workmad3 has quit [Ping timeout: 272 seconds]
<
samkottler>
agreed
havenwood has quit [Read error: Connection reset by peer]
ZachBeta has joined #rubygems
luislavena has joined #rubygems
kyzh has joined #rubygems
kyzh is now known as Guest44586
huoxito has joined #rubygems
davispuh has quit [Ping timeout: 256 seconds]
arohner has quit [Remote host closed the connection]
arohner has joined #rubygems
shaiguit1r has joined #rubygems
shaiguitar has quit [Read error: Connection reset by peer]
almostwhitehat has quit [Ping timeout: 246 seconds]
almostwhitehat has joined #rubygems
havenwood has joined #rubygems
arohner has quit [Remote host closed the connection]
arohner has joined #rubygems
arohner has quit [Remote host closed the connection]
arohner has joined #rubygems
davispuh has joined #rubygems
arohner has quit [Remote host closed the connection]
arohner has joined #rubygems
havenwood has quit [Remote host closed the connection]
arohner has quit [Ping timeout: 265 seconds]
arohner has joined #rubygems
jalessio has quit [Quit: Leaving.]
arohner has quit [Remote host closed the connection]
arohner has joined #rubygems
stevenharman has quit [Ping timeout: 246 seconds]
imperator has quit [Quit: Valete!]
almostwhitehat has quit [Remote host closed the connection]
jkline has joined #rubygems
huoxito has quit [Remote host closed the connection]
havenwood has joined #rubygems
havenwood has quit [Remote host closed the connection]
einarj has quit [Remote host closed the connection]
ZachBeta has quit [Quit: Computer has gone to sleep.]
arohner has quit [Remote host closed the connection]
arohner has joined #rubygems
huoxito has joined #rubygems
arohner has quit [Ping timeout: 272 seconds]
huoxito has quit [Remote host closed the connection]
huoxito has joined #rubygems
huoxito has quit [Read error: Connection reset by peer]
huoxito has joined #rubygems
baburdick has joined #rubygems
huoxito has quit [Remote host closed the connection]
<
dwradcliffe>
evan: can you setup IAM access for me?
<
evan>
I need to get you the creds somehow? do you have OTR IM?
Guest44586 has quit [Remote host closed the connection]
shevy has left #rubygems ["I'll be back ... maybe"]
<
dwradcliffe>
evan: yep
havenwood has joined #rubygems
<
evan>
i'm evan@phx.io on IM
wfarr_ has joined #rubygems
wfarr_ is now known as wfarr
wfarr has quit [Changing host]
wfarr has joined #rubygems
<
evan>
well thats a new one
jonahR has quit [Quit: jonahR]
<
evan>
decryption failed or bad record mac
<
evan>
can't say I've seen that before.
<
wfarr>
yeah, another githubber hit this earlier as well
<
wfarr>
only occurs under the bundler 1.4 rc too
<
evan>
only happens with bundler 1.4.0?
<
wfarr>
regardless of ruby version
<
evan>
good differential there
<
evan>
sounds like something for indirect and friends to look at
<
indirect>
I'm taking a look
<
wfarr>
fwiw, it only started today
<
indirect>
I... don't even know what changed in 1.4 as far as SSL goes
<
wfarr>
I bundled a project with the same deps yesterday, same bundler version
<
drbrain>
evan: wfarr: I think someone on jruby was hitting that too
<
drbrain>
I told them to report it here, but I don't think they have
<
evan>
maybe an error coming from fastly's SSL?
<
wfarr>
vuksan would probably be able to figure out pretty quickly
<
evan>
does it happen everytime?
<
wfarr>
lemme blow away those gems from GEM_PATH so it will refetch them
<
indirect>
wfarr: did you update openssl?
<
indirect>
hmm... or maybe rubygems.org did
<
indirect>
there's a known ubuntu that causes this error :/
<
evan>
fucking OpenSSL.
<
evan>
also kills me that distros fuck with OpenSSL.
<
indirect>
wfarr: are you seeing this with or without -j
<
evan>
oh thats a good call
<
wfarr>
indirect: I have BUNDLE_JOBS set; I can try unsetting that
<
evan>
could a lowlevel socket be being used incorrectly
<
evan>
across multiple SSL sessions
<
indirect>
this seems to be a bug specific to multiprocess or multithreaded installs
<
evan>
that would def cause decryption errors.
<
indirect>
and that would explain why it is 1.4 only
<
indirect>
and openssl isn't threadsafe
<
indirect>
well, it's "threadsafe if you only ever use an ssl connection from one thread"
<
indirect>
which sounds like "not threadsafe" to me
<
wfarr>
trying to confirm jobs vs. not jobs
<
evan>
thats actually a safe constraint
<
evan>
you shouldn't be sharing SSL connections across threads.
<
indirect>
I guess this then becomes a question for drbrain too
<
indirect>
because we're using net::http::persistent
<
indirect>
across threads
<
indirect>
actually on MRI I think the installers fork
<
drbrain>
with NHP a connection isn't shared across threads (at this time)
<
indirect>
okay good
<
indirect>
I'm pretty sure that that would break openssl :)
<
drbrain>
I've had requests for adding thread pooling
<
wfarr>
weird it didn't fail that time
* samkottler
is here now
<
indirect>
drbrain: ah, yeah, that could be nice
luislavena has quit [Remote host closed the connection]
<
drbrain>
is it safe to create an OpenSSL socket on thread A then use it exclusively on a separate thread?
<
samkottler>
reading the backscroll
<
indirect>
drbrain: yes
<
indirect>
Is OpenSSL thread-safe? Yes (with limitations: an SSL connection may not concurrently be used by multiple threads).
luislavena has joined #rubygems
<
drbrain>
then it's just as safe as Net::HTTP is
<
samkottler>
drbrain: you can pass a single connection between threads safely
<
samkottler>
obviously that means you have to stop work on the thread and then pass the connection before starting work again
<
indirect>
this ticket seems to demonstrate that this particular problem is related to multi-process openssl connections :/
<
samkottler>
openssl is such a pile of doody
<
indirect>
the OP has a bunch of forked processes that are each tasked with installing a single gem all cascade fail
<
evan>
indirect: bundler -j uses multiple processes?
<
indirect>
evan: the MRI version forks
<
indirect>
the jruby version has a threadpool
<
evan>
that seems like an odd decision
<
wfarr>
having a hard time reproducing suddenly; really wondering if it's just a nasty race condition
* samkottler
would love to see MRI just use fork(2) if that's not already the case
<
wfarr>
I know skalnik hit this earlier today trying to bundle up some other, different projects today as well
<
indirect>
wfarr: definitely a race condition, the other reporters don't see it every time either
luislavena has quit [Ping timeout: 252 seconds]
<
evan>
indirect: were processes a lot fast than threads on MRI for this?
<
indirect>
evan: it wasn't a decision so much as a japanese rubyist using fork, us adding threads for jruby support, and it seeming to work :P
<
indirect>
I am perfectly okay with doing all threads
<
evan>
this is all IO
<
evan>
starting a bunch of processes doesn't help much
<
evan>
it just balloons the memory footprint.
<
samkottler>
indirect evan: I should have time tomorrow to roll a patch to switch over to threads
<
indirect>
samkottler: awesome
cowboyd has quit [Remote host closed the connection]
<
indirect>
I will see if anyone else bundler-y has time as well
<
evan>
indirect: that could solve this problem
<
samkottler>
indirect: I'll do it next week at the latest since I'm making time off around rubyconf
<
evan>
I worry slightly about share openssl memory blobs across processes
<
evan>
about sharing, rather.
<
samkottler>
evan: creating a new connection per thread should be fine
<
samkottler>
slower, but more safe
<
evan>
you can use a pool fine.
<
evan>
that shouldn't be an issue.
<
evan>
I thought you meant new thread per connection
<
evan>
you said the inverse.
luislavena has joined #rubygems
havenwood has quit [Remote host closed the connection]
havenwood has joined #rubygems
einarj has joined #rubygems
wfarr has quit [Quit: Lost terminal]
einarj has quit [Remote host closed the connection]
luislavena has quit []
<
evan>
lmarburger: poke. I really need fastly to increase those s3 timesout
dwknoxy has quit [Quit: Computer has gone to sleep.]
arohner has joined #rubygems
erichmenge has quit [Ping timeout: 245 seconds]
erichmenge has joined #rubygems
einarj has joined #rubygems
einarj has quit [Ping timeout: 264 seconds]
Vendethi_ has joined #rubygems
Vendethi_ has quit [Ping timeout: 252 seconds]
jkline has quit [Quit: jkline]
knappe has quit [Quit: knappe]
arohner has quit [Remote host closed the connection]