2014-07-17 22:52
closer changed the topic of #ruby-core to: check the latest release candidate for 1.9.1 release ftp.ruby-lang.org:/home/yugui/ruby-1.9.1-r26021+1.tar.bz2
00:17
enebo has quit [Quit: enebo]
00:25
enebo has joined #ruby-core
00:28
charliesome has joined #ruby-core
00:36
enebo has quit [Quit: enebo]
01:05
bffff_ has joined #ruby-core
01:35
travis-ci has joined #ruby-core
01:35
travis-ci has left #ruby-core [#ruby-core]
01:47
Philpax has quit [Read error: Connection reset by peer]
02:02
travis-ci has joined #ruby-core
02:02
travis-ci has left #ruby-core [#ruby-core]
02:12
charliesome has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
02:30
travis-ci has joined #ruby-core
02:30
travis-ci has left #ruby-core [#ruby-core]
02:55
Philpax has joined #ruby-core
06:19
_ht has joined #ruby-core
07:17
charliesome has joined #ruby-core
08:20
Philpax has quit [Quit: Leaving]
08:31
charliesome has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
10:28
travis-ci has joined #ruby-core
10:28
travis-ci has left #ruby-core [#ruby-core]
12:43
nagachika has joined #ruby-core
12:52
nagachika has quit [Remote host closed the connection]
12:53
nagachika has joined #ruby-core
12:59
nagachika has quit [Remote host closed the connection]
12:59
nagachika has joined #ruby-core
13:41
Philpax has joined #ruby-core
14:00
Philpax has quit [Quit: Leaving]
14:10
charliesome has joined #ruby-core
14:17
shinnya has joined #ruby-core
14:31
charliesome has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
14:37
charliesome has joined #ruby-core
14:42
charliesome has quit [Ping timeout: 240 seconds]
14:48
enebo has joined #ruby-core
14:48
travis-ci has joined #ruby-core
14:48
travis-ci has left #ruby-core [#ruby-core]
15:00
charliesome has joined #ruby-core
15:19
charliesome has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
16:55
nurse has quit [Ping timeout: 260 seconds]
16:58
nurse has joined #ruby-core
17:17
nagachika has quit [Remote host closed the connection]
17:17
nagachika has joined #ruby-core
17:22
nagachika has quit [Ping timeout: 255 seconds]
17:46
shinnya has quit [Ping timeout: 255 seconds]
18:24
charliesome has joined #ruby-core
18:25
charliesome has quit [Client Quit]
18:35
kosaki6 has joined #ruby-core
18:35
mpapis has quit [*.net *.split]
18:36
charliesome has joined #ruby-core
18:38
mpapis has joined #ruby-core
18:52
luiz_lha has joined #ruby-core
19:02
charliesome has quit [Ping timeout: 250 seconds]
19:04
charliesome has joined #ruby-core
19:16
charliesome has quit [Ping timeout: 256 seconds]
19:18
charliesome has joined #ruby-core
19:27
charliesome has quit [Ping timeout: 255 seconds]
19:29
charliesome has joined #ruby-core
19:55
drbrain has quit [*.net *.split]
19:55
enebo has quit [*.net *.split]
19:55
drbrain has joined #ruby-core
19:55
enebo has joined #ruby-core
20:00
charliesome has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
21:01
<
headius >
anyone around that could help me understand part of MRI?
21:01
<
headius >
_ko1 or nokada_ perhaps
21:01
<
headius >
basic question...reading through IO#read code, it seems like it's allocating a string using the size requested
21:02
<
headius >
i.e. read(1_000_000_000) allocates a string of 1GB
21:02
<
headius >
either that's not happening and I'm reading it wrong, or it is happening but I can't see it using OS tools
21:11
<
drbrain >
headius: it sure looks like it allocates a 1GB string to me
21:12
<
drbrain >
headius: and I see in Activity Monitor that ruby is using 1GB of memory
21:13
<
drbrain >
allocated in rb_str_modify_expand called from io_setstrbuf
21:21
<
headius >
drbrain: where's the 1GB in activity monitor?
21:21
<
headius >
it's not showing under any of the memory columns I tried
21:22
<
drbrain >
I ran uby -e 'open "/dev/zero" do |io| io.read 1_000_000_000 end; sleep'
21:22
<
drbrain >
+r, of course
21:22
<
drbrain >
and it showed in the Memory column of Activity Monitor
21:23
<
drbrain >
I was trying to set a breakpoint before the read(2)
21:23
<
headius >
ok, I should clarify...I'm reading a fixed-length file
21:23
<
drbrain >
but I'm no good at debuggers
21:23
<
headius >
the logic as written
*seems* like it should still eagerly allocate a 1GB buffer but I can't see it
21:23
<
drbrain >
according to what I'm reading it shouldn't make a difference
21:23
<
headius >
try a small file for me please
21:24
<
headius >
does MRI have something that would re-pack a string down to actual content?
21:24
<
drbrain >
oh, that's probably due to rb_io_set_length right after io_fread
21:24
<
headius >
I looked at that but it doesn't seem to do anything to the buffer
21:26
<
drbrain >
you're right
21:26
<
drbrain >
I have a call in a few minutes before I can look at this more
21:55
<
drbrain >
looks like I need to rebuild ruby
21:56
<
headius >
drbrain: I've got enebo looking too
21:56
<
headius >
one of those days where I feel like I'm blind to some line of code that would explain this
22:12
<
drbrain >
headius: ok
22:12
Philpax has joined #ruby-core
22:12
<
drbrain >
I broke between the REALLOC_N and writing to the string
22:12
<
drbrain >
so it would be the same as a small file
22:12
<
drbrain >
if I look in Activity Monitor it shows:
22:13
<
drbrain >
Real Memory Size 20.6 MB
22:13
<
drbrain >
and VM Size: 3.28 GB
22:13
<
drbrain >
but if I run vmmap against the PID I get:
22:14
<
drbrain >
I'll gist it
22:14
<
drbrain >
so I think this is malloc() tricks
22:15
<
enebo >
It says reserved but does actually do it
22:15
<
enebo >
So if you start writing to that pointer it deals with that and grows it out?
22:16
<
headius >
I figured it would end up being that
22:16
<
enebo >
headius: now emulate that
22:16
<
drbrain >
if I put a sleep after the read and check Activity Monitor its using about 1GB
22:16
<
headius >
yeah right
22:16
_ht has quit [Remote host closed the connection]
22:16
<
headius >
I may have to rethink the ported logic
22:16
<
enebo >
back to #jruby!
22:17
<
headius >
drbrain: that's the gist from reading a small file with read(1G)?
22:17
<
drbrain >
I think line 165 of the gist is most important
22:17
shinnya has joined #ruby-core
22:17
<
drbrain >
I think it would be the same because I haven't hit anything IO related for the breakpoint
22:18
<
drbrain >
well, rb_io_check_closed
22:18
<
headius >
well shuckers
22:19
<
headius >
I can't do that sort of trick because all memory comes from the heap in JVM
22:19
<
drbrain >
btw, 958MB is 1004535808 bytes
22:19
<
drbrain >
so the BYTES ALLOCATED matches
22:19
<
headius >
this is interesting though...a really large string gets no more weight in GC than a small one because they're allocating the store off heap
22:20
<
headius >
I suppose the GC could be looking into the string buffer though and doing some weighting?
22:20
<
drbrain >
it may vary on different malloc() implementations
22:24
<
enebo >
drbrain: thanks for the info. This is pretty fascinating. I am still happy I don’t write C anymore but this is cool :)
22:24
<
drbrain >
I confirmed that a small file has the same behavior
22:25
<
drbrain >
enebo: I think that for large allocations malloc() uses mmap() so the pages are only reserved as address space
22:27
<
drbrain >
and the address map shows the MALLOC_LARGE adjoining regions hold 1000001536 bytes
22:28
<
headius >
yeah, fair enough
22:28
<
enebo >
So MRI did not invent their own alloca and they are using Apple’s default stuff or is this much lower?
22:28
<
headius >
I couldn't find anything custom in MRI around this
22:29
<
drbrain >
if you follow the REALLOC_N macro down it ends up at realloc()
22:29
<
headius >
I think it's just malloc/realloc magic
22:29
<
drbrain >
there's some hooks into the GC to allow the GC to run if that is needed
22:29
<
drbrain >
but ultimately it uses malloc()
22:44
enebo has quit [Quit: enebo]
23:02
luiz_ has joined #ruby-core
23:02
luiz_ is now known as Guest96311
23:03
luiz_lha has quit [Ping timeout: 250 seconds]
23:04
Guest96311 has quit [Max SendQ exceeded]
23:07
luiz_lha has joined #ruby-core
23:22
luiz_lha has quit [Max SendQ exceeded]
23:40
luiz_lha has joined #ruby-core
23:43
luiz_lha has quit [Max SendQ exceeded]
23:44
luiz_lha has joined #ruby-core
23:47
luiz_lha has quit [Max SendQ exceeded]
23:49
luiz_lha has joined #ruby-core
23:52
luiz_lha has quit [Max SendQ exceeded]
23:54
luiz_lha has joined #ruby-core
23:55
hsbt is now known as hsbt_away
23:56
luiz_lha has quit [Max SendQ exceeded]
23:57
luiz_lha has joined #ruby-core
23:59
luiz_lha has quit [Max SendQ exceeded]