ascarter has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
nankyokusei has joined #ruby
mjuszczak has quit []
nankyokusei has quit [Ping timeout: 246 seconds]
<biox>
Hi all! I have a homework assignment: Create a ruby script that takes a huge file (20GB) that contains a number on each line. Supply two arguments: filename and N. Output should be the N largest numbers as specified by the user. Here is what I've come up with so far: https://hastebin.com/basefojoji.vbs - I think it works, but it seems to be unbearably slow. Any advice or direction would be appreciated.
<biox>
It's been hanging for awhile, I'm not sure if the excessive array sorting is causing it to process slowly or if there is a mistake somewhere in my code.
quobo has quit [Quit: Connection closed for inactivity]
Guest69 has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
railswebdev has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
Rr4sT has joined #ruby
eightlimbed has quit [Ping timeout: 246 seconds]
TinkerTyper has quit [Ping timeout: 246 seconds]
bauruine has quit [Ping timeout: 246 seconds]
nesoi has joined #ruby
hutch34 has joined #ruby
jameser has joined #ruby
d^sh has quit [Ping timeout: 260 seconds]
hogetaro has quit [Quit: Leaving...]
d^sh has joined #ruby
bauruine has joined #ruby
uZiel has joined #ruby
elifoster has quit [Ping timeout: 240 seconds]
ascarter has joined #ruby
shinnya has quit [Ping timeout: 240 seconds]
eightlimbed has joined #ruby
ecuanaso has joined #ruby
TinkerTyper has joined #ruby
ascarter has quit [Ping timeout: 246 seconds]
uZiel has quit [Ping timeout: 248 seconds]
MarkBilk_ has joined #ruby
ascarter has joined #ruby
mim1k has joined #ruby
MarkBilk has quit [Ping timeout: 240 seconds]
kies has quit [Ping timeout: 268 seconds]
d^sh has quit [Ping timeout: 255 seconds]
d^sh has joined #ruby
uZiel has joined #ruby
enterprisey has joined #ruby
jameser_ has joined #ruby
ecuanaso has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
troys is now known as troys_
mim1k has quit [Ping timeout: 240 seconds]
jameser has quit [Ping timeout: 240 seconds]
gothicsouth has joined #ruby
ascarter has quit [Read error: Connection reset by peer]
im0nde has quit [Ping timeout: 255 seconds]
Guest69 has joined #ruby
ascarter has joined #ruby
poloych has quit [Remote host closed the connection]
minimalism has joined #ruby
<lostapathy>
biox good luck with the homework. Suggest you add some puts debugging inside the loop to see what's going on and what progress is happening
railswebdev has joined #ruby
uZiel has quit [Ping timeout: 248 seconds]
charliesome has joined #ruby
latemus has joined #ruby
poloych has joined #ruby
Xakotu has joined #ruby
genpaku has quit [Remote host closed the connection]
Xakotu has quit [Client Quit]
genpaku has joined #ruby
gothicsouth has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
eightlimbed has quit [Ping timeout: 246 seconds]
ascarter has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Mia has joined #ruby
Mia has quit [Changing host]
Mia has joined #ruby
MarkBilk__ has joined #ruby
__Yiota has joined #ruby
troys_ is now known as troys
solocshaw has quit [Ping timeout: 246 seconds]
MarkBilk_ has quit [Ping timeout: 255 seconds]
jrafanie has joined #ruby
AnoHito has joined #ruby
Demo318 has joined #ruby
__Yiota has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<biox>
But the problem is that it takes 30 minutes to process a 16GB file, which seems absurd to me.
<biox>
It looks like I may be using readlines, and switching to a different kind of loop might make processing faster. But I cannot understand why a simple sort | tail in bash is orders of magnitude faster than my method
moei has quit [Read error: Connection reset by peer]
d^sh has quit [Ping timeout: 240 seconds]
s2013 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
plujon has quit [Remote host closed the connection]
contradictioned has quit [Ping timeout: 255 seconds]
d^sh has joined #ruby
<darix>
biox: .each_line might be more efficient
johnzorn has joined #ruby
<biox>
darix: I think that's what I'm doing? f.each_line do |line|
<biox>
but I think my problem is that I'm making an array, doing a size comparison against it, and dropping the last value on every single line in the file, which I assume makes it slow.
jameser_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
kgzm has quit [Quit: WeeChat 1.8]
bauruine has quit [Ping timeout: 246 seconds]
bauruine has joined #ruby
webopsx has joined #ruby
cyphase has quit [Ping timeout: 240 seconds]
nupizdets has joined #ruby
Dimik has quit [Ping timeout: 240 seconds]
DeeJayh_ has joined #ruby
cyphase has joined #ruby
DeeJayh has quit [Ping timeout: 240 seconds]
cyphase has quit [Max SendQ exceeded]
cyphase has joined #ruby
AnoHito_ has joined #ruby
Silthias has quit [Ping timeout: 246 seconds]
enterprisey has quit [Remote host closed the connection]
jameser has joined #ruby
AnoHito has quit [Ping timeout: 246 seconds]
gothicsouth has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
charliesome has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
charliesome has joined #ruby
harfangk has joined #ruby
troys has quit [Quit: Bye]
blackwind_123 has joined #ruby
eightlimbed has quit [Ping timeout: 246 seconds]
elifoster has quit [Ping timeout: 240 seconds]
Silthias has joined #ruby
LukyCZ has joined #ruby
Silthias has quit [Ping timeout: 246 seconds]
<LukyCZ>
Hi guys, I've been learning Ruby for some time and still learning, but I ran out of ideas on what to write. Could you suggest me something long but easy ? Thanks
Bock has joined #ruby
latemus has quit [Ping timeout: 268 seconds]
<konsolebox>
try long and difficult
uZiel has joined #ruby
<LukyCZ>
Nah, I'm not that good to do "difficult"
<LukyCZ>
I just want it to be fun
nankyokusei has joined #ruby
Silthias has joined #ruby
<konsolebox>
LukyCZ: you like music? you can write a simple ruby cli that harvests a list of music pathnames from a file and play them randomly using an external cli player, like sox or mplayer. take note of https://ruby-doc.org/core-2.3.1/Kernel.html#method-i-exec.
<tobiasvl>
well right now you don't allow any slashes except the double slashes after protocol
dviola has joined #ruby
<megan_>
right
^mtkd has quit [Ping timeout: 255 seconds]
<matthewd>
tobiasvl: Well, it allows anything after the first character of the hostname
<tobiasvl>
yeah
<tobiasvl>
megan_: spaces included. but you don't match on them. if you put a ^ at the beginning and an $ at the end it will be clearer
<tobiasvl>
your regex can be in the middle of a string, with anything on either side
<elomatreb>
\A and \z are probably what you want, ^/$ match lines, not the entire string
<tobiasvl>
very true
<megan_>
oke oke
<megan_>
Could you try and implement that in "[A-Za-z]:\/\/(www\.)?[A-Za-z0-9]"?
<megan_>
Is that possible?
mtkd has joined #ruby
<elomatreb>
You need to anchor your regex because it would otherwise match anywhere in a string, so "pnfigjdoifj http://a.cool/url asdfasd" would still match
<megan_>
right
jenrzzz has joined #ruby
jenrzzz has joined #ruby
jenrzzz has quit [Changing host]
<megan_>
I really should learn regex a bit more :P
nadir has quit [Quit: Connection closed for inactivity]
fxa90id has joined #ruby
ur5us has joined #ruby
konsolebox has quit [Ping timeout: 240 seconds]
webopsx has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
whiteline has joined #ruby
whiteline has quit [Remote host closed the connection]
hs366 has quit [Quit: Leaving]
dionysus69 has quit [Quit: dionysus69]
dionysus69 has joined #ruby
dviola has quit [Quit: WeeChat 1.8]
alfiemax has joined #ruby
<alfiemax>
hi all
<alfiemax>
can anyone help me with this error when trying to run bundler (using rvm ) in a rails app
<alfiemax>
You must recompile Ruby with OpenSSL support or change the sources in your Gemfile from 'https' to 'http'. Instructions for compiling with OpenSSL using RVM are available at
PresidentBiscuit has joined #ruby
anisha_ has joined #ruby
<matthewd>
alfiemax: You have an error that is describing what's wrong, and even giving you a URL that [it claims] will tell you how to fix it. What help do you need?
<alfiemax>
tried all that...
<alfiemax>
curious thing is the source in gemfile is http
<alfiemax>
and not https
zacts has joined #ruby
anisha has quit [Ping timeout: 260 seconds]
PresidentBiscuit has quit [Ping timeout: 240 seconds]
rohitpaulk has quit [Ping timeout: 255 seconds]
jenrzzz has quit [Ping timeout: 240 seconds]
Beams has joined #ruby
rohitpaulk has joined #ruby
charliesome has joined #ruby
charliesome has quit [Client Quit]
nankyokusei has joined #ruby
xall has joined #ruby
sepp2k has joined #ruby
nankyokusei has quit [Ping timeout: 246 seconds]
mim1k has quit [Ping timeout: 246 seconds]
ujjain has joined #ruby
ujjain has joined #ruby
ujjain has quit [Changing host]
brent__ has joined #ruby
patarr has joined #ruby
JoshS has quit [Ping timeout: 246 seconds]
JoshS has joined #ruby
polysics has joined #ruby
sarbs has quit [Ping timeout: 246 seconds]
brent__ has quit [Ping timeout: 255 seconds]
mim1k has joined #ruby
patarr has quit [Ping timeout: 240 seconds]
Burgestrand has quit [Quit: Closing time!]
Puffball has quit [Remote host closed the connection]
sarbs has joined #ruby
anisha_ has quit [Quit: This computer has gone to sleep]
Burgestrand has joined #ruby
psychicist__ has joined #ruby
Puffball has joined #ruby
anisha_ has joined #ruby
bvcosta has joined #ruby
xall_ has joined #ruby
cfec0b8d has joined #ruby
Burgestrand has quit [Ping timeout: 260 seconds]
xall has quit [Ping timeout: 240 seconds]
Terens has joined #ruby
<Terens>
hell/o
<Terens>
should I use require in my gems or autoload ?
xall_ has quit [Read error: Connection reset by peer]
<elomatreb>
Generally explicit requires are preferred
<dminuoso>
^^^
<dminuoso>
Terens: But if you prefer, there's nothing inherently wrong with autoload
xall_ has joined #ruby
rohitpaulk has quit [Ping timeout: 240 seconds]
<canton7>
autoload was being discouraged a few years ago - mutterings of races, and it eventually being phased out. Has that changed?
<Terens>
I eyeballed actioncable code and it was like 100% autoload thats why
cout has quit [Remote host closed the connection]
<Terens>
I am asking
<elomatreb>
I've done %w[a lot of gems].each {|g| require g } if I had many gems
<apeiros>
canton7: races have been eliminated, matz still doesn't like it and wants it out (have that on personal account)
saschavoid has joined #ruby
<canton7>
cool, good to know thanks
<apeiros>
I don't remember his reasoning, but iirc it was along the lines of having a fully loaded system early vs. never really knowing whether all is loaded yet (and what you can make if it when a system is fully loaded, probably wrt optimizations)
nankyokusei has joined #ruby
joonty has joined #ruby
hutch34 has joined #ruby
conta has quit [Read error: Connection reset by peer]
modin has joined #ruby
xall_ has quit [Quit: xall_]
conta has joined #ruby
hutch34 has quit [Ping timeout: 255 seconds]
sleetdrop has joined #ruby
psychicist__ has quit [Quit: leaving]
psychicist__ has joined #ruby
jameser has quit [Ping timeout: 260 seconds]
nankyokusei has quit [Remote host closed the connection]
xall has joined #ruby
xall has quit [Client Quit]
xall has joined #ruby
nickjj has joined #ruby
mtkd has quit [Ping timeout: 260 seconds]
mtkd has joined #ruby
alfiemax has quit [Remote host closed the connection]
patarr has joined #ruby
xall has quit [Read error: Connection reset by peer]
xall has joined #ruby
Burgestrand has joined #ruby
psychicist__ has quit [Ping timeout: 246 seconds]
im0nde has joined #ruby
glcx has quit [Quit: This computer has gone to sleep]
patarr has quit [Ping timeout: 260 seconds]
Fernando-Basso has joined #ruby
Mia has joined #ruby
Mia has joined #ruby
Mia has quit [Changing host]
astrobunny has quit [Remote host closed the connection]
astrobunny has joined #ruby
axyjo has quit [Read error: Connection reset by peer]
mim1k has quit [Read error: Connection reset by peer]
mim1k has joined #ruby
astrobunny has quit [Ping timeout: 255 seconds]
cyphase has quit [Ping timeout: 240 seconds]
Puffball has quit [Remote host closed the connection]
AnoHito_ has quit [Quit: Leaving]
Puffball has joined #ruby
joonty has quit [Quit: This computer has gone to sleep]
AnoHito has joined #ruby
cyphase has joined #ruby
mim1k has quit [Ping timeout: 268 seconds]
catphish has joined #ruby
joonty has joined #ruby
quobo has quit [Quit: Connection closed for inactivity]
mim1k has joined #ruby
gregf_ has joined #ruby
birdyondrugs has quit [Ping timeout: 240 seconds]
Doow has joined #ruby
wnd has quit [Excess Flood]
wnd has joined #ruby
Asher has quit [Quit: Leaving.]
<duper>
If I have an array of Hashes like { :name => 'abc', :value => 'xyz' } .. then how do I condense so there's only one entry for each :name and :value becomes Array#String (by combining all the strings for existing :name's) ?
<duper>
almost sounds so easy that you'd think there would be an existing Hash or Array instance_method that does this
Burgestrand has quit [Remote host closed the connection]
Burgestrand has joined #ruby
Doow has quit [Quit: Leaving]
govg has quit [Ping timeout: 240 seconds]
bvcosta has quit [Remote host closed the connection]
bvcosta has joined #ruby
ecuanaso has joined #ruby
hogetaro has joined #ruby
bvcosta has quit [Remote host closed the connection]
govg has joined #ruby
Silthias has quit [Ping timeout: 260 seconds]
Silthias has joined #ruby
<Terens>
hi
govg has quit [Ping timeout: 240 seconds]
<Terens>
having a method def errback &block
<Terens>
and use it like errback { code }
<Terens>
how can I pass a method?
<Terens>
def mymethod end then errback &:mymethod does nothing
<elomatreb>
If you don't yield to the block or call it explicitely it won't be executed
PresidentBiscuit has joined #ruby
whiteline has joined #ruby
PresidentBiscuit has quit [Ping timeout: 246 seconds]
ur5us has quit [Remote host closed the connection]
<Burgestrand>
Terens you can do `errback(&method(:mymethod))` or just do `errback { mymethod() }`
Burgestrand has quit [Remote host closed the connection]
Silthias1 has joined #ruby
<Terens>
I did the second one lol
Silthias has quit [Ping timeout: 246 seconds]
Burgestrand has joined #ruby
GinoMan has quit [Ping timeout: 240 seconds]
charliesome has joined #ruby
uZiel has quit [Ping timeout: 248 seconds]
username1 has joined #ruby
glcx has joined #ruby
glcx has joined #ruby
glcx has quit [Changing host]
GinoMan has joined #ruby
synthroid has joined #ruby
Mortomes|Work has quit [Ping timeout: 260 seconds]
mim1k has quit [Read error: Connection reset by peer]
username1 has quit [Ping timeout: 260 seconds]
helpa has quit [Remote host closed the connection]
Hunter1 has quit [Remote host closed the connection]
Hunter1 has joined #ruby
uZiel has joined #ruby
cdg has joined #ruby
arup_r has quit [Ping timeout: 240 seconds]
pupsicle has quit [Remote host closed the connection]
Asher has quit [Quit: Leaving.]
jphase has joined #ruby
uZiel has quit [Ping timeout: 248 seconds]
aenialis_ has joined #ruby
vondruch has quit [Ping timeout: 246 seconds]
__Yiota has joined #ruby
uZiel has joined #ruby
Spydar has joined #ruby
Spydar is now known as Guest73007
cdg has quit [Remote host closed the connection]
PatrikasZvaigzde has quit [Ping timeout: 260 seconds]
PatrikasZvaigzde has joined #ruby
SpydarOO7 has quit [Disconnected by services]
Guest73007 has quit [Changing host]
Guest73007 has joined #ruby
Guest73007 is now known as SpydarOO7
charliesome has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
govg has joined #ruby
gothicsouth has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
spectrum has quit [Ping timeout: 240 seconds]
<LyndsySimon>
I know Ruby has the concept of "autoload", and that Rails has a similar concept and builds on it with "eager loading". I'm looking for a way to eager load all files in a directory, preferably without hooking into Rails' `config.eager_load_paths`, since I want it to happen even when Rails' eager loading is turned off (i.e., in local development).
<LyndsySimon>
You know what? Scratch that. I was trying to strip out part of a complex problem I'm working on, and I think I just figured out a way to avoid it entirely.
<LyndsySimon>
I'm still interested if someone has an answer, but it's not blocking me anymore :)
__Yiota has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Asher has joined #ruby
Burgestrand has quit [Ping timeout: 255 seconds]
lagweezle is now known as lagweezle_away
__Yiota has joined #ruby
rohitpaulk has quit [Ping timeout: 246 seconds]
ta_ has quit [Remote host closed the connection]
Burgestrand has joined #ruby
megan_ has quit [Quit: Page closed]
Burgestrand has quit [Remote host closed the connection]
aenialis has left #ruby ["Good Bye"]
shinnya has joined #ruby
ledestin has joined #ruby
al2o3-cr has quit [Ping timeout: 255 seconds]
aenialis_ is now known as aenialis
ur5us has joined #ruby
spectrum has joined #ruby
spectrum has quit [Read error: Connection reset by peer]
ur5us has quit [Ping timeout: 268 seconds]
blackwind_123 has quit [Ping timeout: 240 seconds]
Nick__ has quit [Remote host closed the connection]
chouhoulis has joined #ruby
svkurowski has quit [Ping timeout: 240 seconds]
cout has joined #ruby
ledestin has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
svkurowski has joined #ruby
Fernando-Basso has quit [Quit: WeeChat 1.8]
im0nde has quit [Ping timeout: 246 seconds]
Fernando-Basso has joined #ruby
hutch34 has joined #ruby
hutch34 has quit [Ping timeout: 255 seconds]
jrafanie has joined #ruby
nankyokusei has joined #ruby
im0nde has joined #ruby
nankyokusei has quit [Ping timeout: 240 seconds]
catphish has left #ruby ["Leaving"]
gizmore has joined #ruby
aenialis is now known as aenialis_
anisha_ has quit [Quit: This computer has gone to sleep]
aenialis_ has quit [Quit: ChatZilla 0.9.93 [Firefox 54.0/20170612121707]]
yeticry has quit [Quit: leaving]
yeticry has joined #ruby
patarr has joined #ruby
theunraveler has joined #ruby
Mia has quit [Read error: Connection reset by peer]
S007 has joined #ruby
SpydarOO7 has left #ruby [#ruby]
<ljarvis>
LyndsySimon: isn't eager loading just requiring? -- it's a rails thing because it needs to bypass their lazy/auto loading functionality
<ljarvis>
so just require all files in a directory and you'll have access to the contents
ruby-lang559 has joined #ruby
antgel has joined #ruby
Mia has joined #ruby
Mia has joined #ruby
Mia has quit [Changing host]
grymmjack has joined #ruby
_mak_ has joined #ruby
aupadhye has quit [Ping timeout: 260 seconds]
<_mak_>
Hi all
<ytti>
</win 20
gothicsouth has joined #ruby
<LyndsySimon>
ljarvis: Is there any easier way to "require all files in a directory" other than enumerating them and building the require statements dynamically?
ruby-lang559 has quit [Quit: Page closed]
gj_ has joined #ruby
muelleme has joined #ruby
quobo has joined #ruby
<_mak_>
is there a software pattern for this use case: I want to 'tag' entities based on the state of some properties, for example, if the entity is of the color red and size > 3 then I want to call it 'big_red'. Some other subsystem will be dealing with entities that are big_red. The question is: Should 'big_red' become a property of that entity or should 'big_red' be the return of a function called 'get_entity_tag()'? Where do we draw the line?
bauruine has quit [Ping timeout: 255 seconds]
rippa has joined #ruby
webopsx has joined #ruby
grymmjack has quit [Ping timeout: 240 seconds]
gj_ is now known as grymmjack
webopsx has quit [Max SendQ exceeded]
mim1k has quit [Read error: Connection reset by peer]
cschneid_ has joined #ruby
cschneid_ has quit [Remote host closed the connection]
cschneid_ has joined #ruby
im0nde has quit [Ping timeout: 240 seconds]
bauruine has joined #ruby
s2013 has joined #ruby
mikecmpbll has quit [Quit: inabit. zz.]
mikecmpbll has joined #ruby
<ljarvis>
LyndsySimon: no, but it's only a one liner. How easier do you want it? :)
<LyndsySimon>
ljarvis: You don't happen to have that one-liner handy, do you? My head is full of other context and I'd like at least document this.
<elomatreb>
Dir["**/*.rb"].each {|f| require f } or something like that
<ljarvis>
LyndsySimon: Dir["my/dir/*.rb"].each { |f| require f }
<ljarvis>
heh
<LyndsySimon>
Woot, thank s:)
<ljarvis>
yes, elomatreb's example will go into subdirectories also
<elomatreb>
Oh, misread that
<LyndsySimon>
Ah, cool, it supports pattern matching then. Even better.
* ljarvis
waits for someone obtuse to offer each(&method(:require))
hutch34 has joined #ruby
grymmjack has quit [Quit: Going offline, see ya! (www.adiirc.com)]
<elomatreb>
Now I want to benchmark &method vs an explicit block
cschneid_ has quit [Read error: Connection reset by peer]
cschneid_ has joined #ruby
yeticry_ has joined #ruby
Hunter1 has quit [Remote host closed the connection]
yeticry has quit [Read error: Connection reset by peer]
Hunter1 has joined #ruby
mim1k has joined #ruby
im0nde has joined #ruby
PatrikasZvaigzde has quit [Quit: Leaving]
aufi has quit [Quit: Leaving]
<biox>
Hello everyone. I have an assignment that requires me to: provide two inputs to a ruby file (filename and N). Assuming the file contains newline-separated integers, parse the file and print the largest N numbers. The filesize I'm operating against is very large (16GB) so I can't load it all in memory at once. To speed things up I'm trying to use the Parallel gem, but once I run the code all of the memory on my
<biox>
machine (10+GB) is consumed. This is true even with a smaller filesize (100MB) so there must be something wrong with my code. Am I missing something? https://hastebin.com/wexipuhixi.vbs
conta has quit [Ping timeout: 260 seconds]
s2013 has quit [Read error: Connection reset by peer]
chalkmonster has joined #ruby
roshanavand has quit [Quit: roshanavand]
<ropeney>
biox, is it the same if you don't sort?
<biox>
ropeney: it is. In fact it's the same if all I do nothing inside of the parallel block
<biox>
so my guess is that Parallel creates a thread from each line first, then fires off the workers or something.
<lostapathy>
biox I thought you had this sorted out yesterday?
uZiel has quit [Ping timeout: 248 seconds]
<ropeney>
I think line isn't being collected by the GC
<biox>
lostapathy: I got it working without parallel operations, I'm trying to make it faster now - running into this new problem
BTRE has quit [Ping timeout: 255 seconds]
<biox>
ropeney: How could I test something like that? The extent of my debugging currently is running stuff in IRB/puts'ing variables, but those methods don't apply here
texasmynsted has quit [Ping timeout: 240 seconds]
aenialis has joined #ruby
<ropeney>
You can look if the Object Ref count keeps climbing
ascarter has joined #ruby
<biox>
ropeney: in my case it sounds like that would be array.reference_count or line.reference_count
<ropeney>
I'm not too sure if that's the case, as they should be dangling references at each reference, waiting to be GC'd
<ropeney>
at each iteration*
enterprisey has joined #ruby
cschneid_ has quit [Remote host closed the connection]
__Yiota has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<ropeney>
biox, watching `GC.stat` might give you some indication if it's the right path.
__Yiota has joined #ruby
muelleme has quit [Ping timeout: 260 seconds]
<biox>
ropeney: Okay great, I will give this a test and get back to you.
__Yiota has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
brent__ has joined #ruby
rohitpaulk has joined #ruby
hobodave has joined #ruby
haylon has joined #ruby
texasmynsted has joined #ruby
__Yiota has joined #ruby
rohitpaulk has quit [Ping timeout: 260 seconds]
chalkmonster has quit [Quit: Daddy's flown, 'cross the ocean.]
ledestin has joined #ruby
<matthewd>
biox: If you're memory constrained, doing several things in parallel seems unlikely to improve your situation
gothicsouth has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
MarkBilk has joined #ruby
ur5us has joined #ruby
mark_66 has quit [Remote host closed the connection]
MarkBilk_ has quit [Ping timeout: 246 seconds]
TomyWork has quit [Ping timeout: 240 seconds]
dc4 has quit [Ping timeout: 240 seconds]
ledestin has quit [Ping timeout: 268 seconds]
dionysus69 has quit [Ping timeout: 240 seconds]
synthroid has quit [Remote host closed the connection]
marxarelli has joined #ruby
ur5us has quit [Ping timeout: 258 seconds]
<biox>
matthewd: While I can't load the huge file into memory all at once, I figured that I could chop it up using parallel threads. The original version of my program used maybe 1GB of memory for a 16GB file. With parallelism, 16GB of memory is consumed completely by a 100MB file. To me this is indicative of a different problem.
<biox>
(The original version of my program being single-threaded, and ran at 100% of 1 core)
<ropeney>
16GB for the 100mb file? :\
hxegon has quit [Quit: leaving]
synthroid has joined #ruby
<biox>
that's right
<biox>
it consumed all 16GB of memory on my local system trying to run against a 100MB file. :/
<matthewd>
biox: You're obviously free to do whatever you like; I can only strongly suggest that you're going down the wrong path
<biox>
matthewd: What path would you recommend if I wanted to improve performance?
<biox>
I mean without using GNU sort ;)
<matthewd>
Think about what's happening inside the loop [of the non-parallel version]
<matthewd>
That loop gets executed for every row in the source file, and there are a lot of those, so it would be advisable to minimize the amount of work therein
gothicsouth has joined #ruby
hosttor has joined #ruby
<biox>
matthewd: Inside of the loop I take each line and toss it into an array. Since I don't have the memory available to toss everything into the array at once my thought was I would periodically sort and trim the array if it grew over a certain size. I found that increasing this size could improve performance slightly, but not greatly (degrading performance at around 10,000,000 array items). Otherwise I can't think
<biox>
of a step I can trim out of the process.
gizmore has quit [Read error: Connection reset by peer]
<biox>
Oh wait, what if I just threw out numbers lesser than array[9] instead of having to sort them
texasmynsted has left #ruby [#ruby]
BTRE has joined #ruby
uZiel has joined #ruby
ferr1 has quit [Quit: WeeChat 1.8]
<matthewd>
That does sound like less work
<biox>
Huh, I overlooked the heck out of that. One sec, I'll get back to you guys - I'll put a hold on the parallel idea.
__Yiota has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
username has joined #ruby
<ropeney>
lol, I was fixated on why you're using so much memory; not how to avoid it all together :(
yehowyada has joined #ruby
rgr has quit [Ping timeout: 246 seconds]
PaulCapestany has quit [Quit: .]
hosttor has quit [Ping timeout: 240 seconds]
TomyLobo has joined #ruby
gdonald has joined #ruby
PaulCapestany has joined #ruby
gusrub has joined #ruby
dc4 has joined #ruby
pupsicle has joined #ruby
__Yiota has joined #ruby
skweek has joined #ruby
jrafanie has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<biox>
I don't think that I can make the comparison any more efficient than if (number is smaller than the smallest number in the array) then do nothing
mim1k has joined #ruby
psychicist__ has joined #ruby
dc4 has quit [Ping timeout: 255 seconds]
<matthewd>
Yeah, that's sounding better. You could probably get slightly more out of the inner part, but bounded memory + a sane execution time sound like a fair improvement as is.
im0nde has quit [Ping timeout: 255 seconds]
SeepingN has joined #ruby
aenialis has quit [Quit: leaving]
PatrikasZvaigzde has joined #ruby
ecuanaso has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
centrx has joined #ruby
centrx has quit [Changing host]
centrx has joined #ruby
ecuanaso has joined #ruby
elifoster has joined #ruby
Beams has quit [Quit: .]
cadillac__ has joined #ruby
blackwind_123 has quit [Ping timeout: 260 seconds]
synthroi_ has joined #ruby
glcx has quit [Quit: Quitte]
gothicsouth has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
antgel has quit [Ping timeout: 240 seconds]
synthroid has quit [Ping timeout: 246 seconds]
plujon has joined #ruby
bvcosta has joined #ruby
elifoster has quit [Ping timeout: 246 seconds]
mim1k has quit [Ping timeout: 260 seconds]
Mia has quit [Ping timeout: 260 seconds]
unreal has joined #ruby
<biox>
matthewd: given that this is already quite fast, wouldn't it make sense to make the process parallel at this point? the memory would still be bounded (I think) and four-cores could result in a significant improvement over a single core if I'm thinking about this correctly.
creshiem has joined #ruby
nitric has joined #ruby
<biox>
I mean clearly Ruby isn't the right tool for doing this quickly, I just want to see what it can do. :)
<ropeney>
I don't think you'll get anything from parallel
<ropeney>
your mutexes are going to prevent most
mikecmpbll has quit [Quit: inabit. zz.]
ecuanaso has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<ropeney>
if you go down parallel, split it in 4 with 4 different complete arrays. Then merge at end :)
<ropeney>
so your inner loops aren't sharing a single array.
<matthewd>
biox: The challenge I'd anticipate is in usefully parallelizing the work... it should now be mostly I/O
Neptu has quit [Ping timeout: 255 seconds]
<matthewd>
I guess you could divide the whole file into chunks, then scan each of those in parallel... likely more work than I'd personally bother with, without a concrete performance target
gothicsouth has joined #ruby
<biox>
matthewd: Sure, and I don't have that target - but I'll give it a shot for fun. The Parallel library is having a hayday with my memory still, so I'll have to work through that. I'll give concurrent-ruby a try as suggested by ropeney as well.
<ropeney>
Instant must always be the target.
<biox>
ropeney: `sort #{filename} | head -n #{selection}`
ecuanaso has joined #ruby
<ropeney>
have you looked at the algorithm they use?
<ropeney>
It is open source.
<biox>
I have trouble following C code :/
<biox>
I tried yesterday and couldn't figure it out
<ropeney>
That's a fair point lol
ryzokuken has joined #ruby
<biox>
The people in #bash say that sort loads the file into memory (preserving maybe 100MB and never dipping into swap intentionally) - then it makes files in /tmp for the rest of the data, so it definitely loads everything at once, does a mergesort (i think) and then prints the output
<ropeney>
Yeah looks like it merge sorts
duderonomy has quit [Ping timeout: 240 seconds]
<ropeney>
and that'd give more accuracy then your code right?
<biox>
I have no idea
<ropeney>
youre one would break if first number was 0?
<biox>
Hah, good point
<biox>
Yeah, a mergesort would probably be good - I looked at a few ruby implementations but they all use arrays, and I can't hold the whole file in an array at once sadly
<ropeney>
your whole file isn't unique
<ropeney>
I hope...
<biox>
It is not. Does Ruby automatically compress stuff before it enters memory?
<biox>
That question is probably way too vague
<ropeney>
I'd merge sort in from each line
Neptu has joined #ruby
<ropeney>
should be rather fast still
<ropeney>
that'd drop duplicates
<biox>
Oh interesting
<biox>
I'll give that a shot next and compare it to my current code
<ropeney>
you of course have to check if there is a duplicate
<biox>
sure, it sounds like that'd require a lot of computation since it'd have to happen on every number
<ropeney>
well, yours happens every number
<ropeney>
it compares
<ropeney>
except yours probably isn't giving the result you want
<biox>
it does, but it compares against a var that's already set versus the mergesort method that would have to look for a matching number on each iteration, unless I'm understanding my code wrong
joonty has quit [Quit: This computer has gone to sleep]
<ropeney>
I think we have different goals is all
mtkd has quit [Ping timeout: 240 seconds]
<ropeney>
I'd rather a solution that gave the right answer, then didn't :P
<ropeney>
if you have 0 at line 1000, rest of your file is useless
mtkd has joined #ruby
ecuanaso has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
hotpancakes has joined #ruby
cbyrda has joined #ruby
<biox>
Yea, I'm also not accounting for negative numbers, really making a lot of assumptions about the input
vuoto has joined #ruby
<ropeney>
all depends on your goal
<ropeney>
i'd load the file in chunks, do the loop with 4 seperate arrays in parallel using merge sort, then merge sort them all together at end.
Neptu has quit [Ping timeout: 255 seconds]
<ropeney>
4 being the amount of threads you want
<biox>
I have never done those things before, so it'd probably be a good practice for me to try. I'll give it a shot and let you know how it turns out.
<ropeney>
I may be passed out before you finish, but goodluck!
Silthias1 has quit [Ping timeout: 246 seconds]
<biox>
And also, thanks for all of your patience! (and you too matthewd)
AndBobsYourUncle has joined #ruby
Silthias has joined #ruby
ecuanaso has joined #ruby
cdg has joined #ruby
saschavoid has quit [Quit: leaving]
cdg_ has joined #ruby
lxsameer has quit [Quit: WeeChat 1.7]
rohitpaulk has joined #ruby
ivanf has joined #ruby
Dimik has joined #ruby
cdg has quit [Ping timeout: 260 seconds]
gothicsouth has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
gusrub has quit [Remote host closed the connection]
gusrub has joined #ruby
gusrub has quit [Remote host closed the connection]
gusrub has joined #ruby
creshiem has quit [Quit: WeeChat 1.8]
eightlimbed has joined #ruby
hobodave has quit [Quit: Computer has gone to sleep.]
ta_ has joined #ruby
Neptu has joined #ruby
jrafanie has joined #ruby
hotpancakes has quit [Remote host closed the connection]
hotpancakes has joined #ruby
bauruine has quit [Max SendQ exceeded]
cadillac__ has quit [Ping timeout: 246 seconds]
AndBobsYourUncle has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
hotpancakes has quit [Ping timeout: 260 seconds]
renchan has quit [Quit: Leaving...]
ecuanaso has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
konsolebox has joined #ruby
jaruga_________ has quit [Ping timeout: 240 seconds]
bauruine has joined #ruby
blackwind_123 has joined #ruby
hutch34 has quit [Ping timeout: 240 seconds]
hotpancakes has joined #ruby
hotpanca_ has joined #ruby
cadillac__ has joined #ruby
duderonomy has joined #ruby
AndBobsYourUncle has joined #ruby
hobodave has joined #ruby
hotpancakes has quit [Ping timeout: 240 seconds]
FastJack has quit [Read error: Connection reset by peer]
FastJack has joined #ruby
omab has joined #ruby
minimalism has joined #ruby
haylon has quit []
cadillac__ has quit [Ping timeout: 255 seconds]
hotpanca_ has quit [Remote host closed the connection]
tubuliferous has joined #ruby
hotpanca_ has joined #ruby
uZiel has joined #ruby
ahrs has quit [Read error: Connection reset by peer]
ur5us has joined #ruby
ahrs has joined #ruby
jaruga_________ has joined #ruby
dangerousdave has joined #ruby
<dangerousdave>
Is there an elegant way to strip newlines when reading a file using File readlines?
marxarelli is now known as marxarelli|afk
<havenwood>
dangerousdave: On 2.4: File.readlines(filename, chomp: true)
<dangerousdave>
havenwood many thanks!
ur5us has quit [Ping timeout: 246 seconds]
<havenwood>
dangerousdave: you're welcome!
skweek has quit [Ping timeout: 240 seconds]
jrafanie has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
ResidentBiscuit has quit [Quit: Critical Miss]
marxarelli|afk is now known as marxarelli
vuoto has quit [Remote host closed the connection]
hotpanca_ has quit [Remote host closed the connection]
dc4 has joined #ruby
cfec0b8d has joined #ruby
bvcosta has quit [Remote host closed the connection]
hutch34 has joined #ruby
bvcosta has joined #ruby
Tonyt_ has joined #ruby
gothicsouth has joined #ruby
sekmo has joined #ruby
bvcosta has quit [Ping timeout: 240 seconds]
hutch34 has quit [Ping timeout: 255 seconds]
hotpanc__ has joined #ruby
pjmelling has joined #ruby
<Tonyt_>
Hi community. I am a noob with Ruby and am teaching myself, but I want to know if there is a roadmap of subject matter. What should I learn first, second, third etc.
<Tonyt_>
Any input?
jason^ has joined #ruby
jrafanie has joined #ruby
workmad3 has quit [Ping timeout: 260 seconds]
<dangerousdave>
Tonyt_ why have you decided to learn Ruby?
<jason^>
occasionally i want to join an array but sometimes it's nil, is there an easy way to return an empty string or join an array inline without having to add a bunch of checks or modifying code to set a var to an empty array?
<havenwood>
Tonyt_: I can't think of a particular order other than avoiding spending any time on class variables or protected.
<Tonyt_>
In order to make web apps, combined with the flatter learning curve
gusrub has quit [Remote host closed the connection]
<dangerousdave>
Tonyt_ do you have experience with other languages?
Rodya_ has joined #ruby
<pjmelling>
Tonyt_: the book Programming Ruby 1.0 & 2.0 (4th ed.) is a great primer and reference
<havenwood>
Tonyt_: I'd suggest first learning some Ruby. Then create a bare bones Rack app so you understand how that works. Then convert your Rack app to Roda or Sinatra. Then maybe try a Rails tutorial.
<Tonyt_>
I would say I am a beginner with Python as well
<dangerousdave>
Tonyt_ For me, I like to learn by solving a real problem, helps keep me interested. I would suggest you create a simple application, and learn Ruby in the process. Perhaps a blog
<havenwood>
Tonyt_: Please feel free to ask here if you have any questions as you go along!
<Tonyt_>
<pjmelling> thank you all
<Tonyt_>
I appreciate the feedback
davidmichaelkarr has joined #ruby
<havenwood>
jason^: Can you #compact the Array before joining?
ych14_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<kspencer>
confused on why I'm getting an 'incompatible library version' for the mysql gem, I know it says 'incompatible' but I don't see any other versions of the libmysqlclient-dev
eightlimbed has joined #ruby
<matthewd>
kspencer: You may need to recompile the gem if you've updated the library package
<matthewd>
kspencer: gem pristine mysql (or mysql2, maybe more likely?)
<kspencer>
welp lets stop the app and see if it works
zeddd has quit [Remote host closed the connection]
ta_ has joined #ruby
<kspencer>
I'm also rvm it seems, does that do anything different?
<kspencer>
using rvm*
ur5us has joined #ruby
<kspencer>
@ matthewd
<matthewd>
Shouldn't
tastygradient has joined #ruby
Murda has quit [Quit: Murda]
tastygradient has quit [Client Quit]
ur5us has quit [Ping timeout: 268 seconds]
marxarelli is now known as marxarelli|afk
ahrs has quit [Remote host closed the connection]
ahrs has joined #ruby
marxarelli|afk is now known as marxarelli
muelleme_ has joined #ruby
muelleme has quit [Ping timeout: 260 seconds]
ben______ has joined #ruby
jenrzzz has quit [Ping timeout: 240 seconds]
ben__ has quit [Ping timeout: 240 seconds]
shinnya has joined #ruby
Rodya__ has joined #ruby
hightower2 has quit [Ping timeout: 240 seconds]
<zenspider>
rawr
sekmo has joined #ruby
<zenspider>
jason^: it'd be better to not have nils to begin with. always an anti-pattern
<zenspider>
"Shouldn't" :P
gothicsouth has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
poloych has joined #ruby
Mia has joined #ruby
Mia has joined #ruby
Mia has quit [Changing host]
lxsameer has joined #ruby
Rodya__ has quit [Client Quit]
Rodya_ has quit [Ping timeout: 258 seconds]
Danielss89 has quit [Quit: Bye]
Danielss89 has joined #ruby
ecuanaso has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
vuoto has joined #ruby
Cohedrin has quit [Read error: Connection reset by peer]
hightower2 has joined #ruby
Cohedrin has joined #ruby
acalycine has joined #ruby
<patarr>
Is there something like a Range but has no upper bound?
<patarr>
e.g. (50...)?
dc4 has quit [Ping timeout: 240 seconds]
<apeiros>
50..Float::INFINITY
* patarr
groans
<patarr>
Okay thanks apeiros
<apeiros>
o0
<apeiros>
groan?
jrafanie has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<patarr>
There is no infinity in computers, so I don't like its explicit reference. Idk just seems weird to me.
<apeiros>
o0
<patarr>
And why does Float only contain this constant, as opposed to Integer as well or something.
<apeiros>
because there does not exist an infinity in the domain of integers, but in computers and the domain of rationals (which floats try to approximate)
rippa has quit [Quit: {#`%${%&`+'${`%&NO CARRIER]
dionysus69 has joined #ruby
hightower2 has quit [Ping timeout: 255 seconds]
jrafanie has joined #ruby
bvcosta has joined #ruby
__Yiota has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
ascarter has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
________ has joined #ruby
ascarter has joined #ruby
poloych has quit [Remote host closed the connection]
<havenwood>
patarr: There's also: 50.step
jrafanie has quit [Client Quit]
<havenwood>
>> 50.step.size
<ruby[bot]>
havenwood: I'm terribly sorry, I could not evaluate your code because of an error: NoMethodError:undefined method `[]' for nil:NilClass
fxa90id has quit [Quit: Leaving]
<havenwood>
#=> Infinity
Bock has joined #ruby
<havenwood>
>> 50.step.first(3)
Bock has quit [Max SendQ exceeded]
<havenwood>
#=> [50, 51, 52]
<ruby[bot]>
havenwood: I'm terribly sorry, I could not evaluate your code because of an error: OpenURI::HTTPError:500 Internal Server Error
<zenspider>
ugh
bvcosta has quit [Ping timeout: 268 seconds]
<havenwood>
Bad bot.
<havenwood>
?cookie ruby[bot]
<ruby[bot]>
ruby[bot]: here's your cookie: 🍪
synthroi_ has quit []
Bock has joined #ruby
async_pr1 has quit [Quit: WeeChat 1.7.1]
async_prince has joined #ruby
MrBusiness has quit [Ping timeout: 246 seconds]
<havenwood>
patarr: You can add #lazy for lazy enumeration so it doesn't take infinite time.
enterprisey has quit [Remote host closed the connection]
mtkd has quit [Read error: Connection reset by peer]
mtkd has joined #ruby
nankyokusei has quit [Ping timeout: 240 seconds]
dionysus69 has quit [Ping timeout: 240 seconds]
guymanndude has joined #ruby
<guymanndude>
hi, I'm working on a Thor::Group script but am having trouble writing tests for it; are there any resources explaining how to approach testing a Thor::Group script with RSpec or something similar?
__Yiota has joined #ruby
<guymanndude>
basically I'm shelling out to an external application and moving and manipulating files
mahlon has quit [Ping timeout: 240 seconds]
mahlon has joined #ruby
marxarelli|afk is now known as marxarelli
Jayson_Virissimo has joined #ruby
tubuliferous has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
biberu has quit []
pupsicle has quit [Quit: Leaving]
zacts has quit [Ping timeout: 255 seconds]
hotpancakes has quit [Remote host closed the connection]
t-recx has quit [Quit: t-recx]
<zenspider>
guymanndude: I don't use thor for anything... but when I'm writing tests for this type of thing I just verify that I'm generating the right command strings/arrays, usually by trapping system directly
<guymanndude>
ok zenspider, thanks. I guess I can create an instance of my script, call it to manipulate the files and check if they've been created and look right
<zenspider>
that is the exact *opposite* of what I just described.