<yorickpeterse>
I suspect that said object gets passed to Rubby, while at the same time being collected. This is a bit odd, since it was just allocated, but I guess it's possible
<yorickpeterse>
oh hm, seems Object also has a bunch of allocate methods
<yorickpeterse>
Perhaps I'm calling my tracer in the wrong place
byprdct_ has quit []
whatasunnyday has quit [Quit: whatasunnyday]
diegoviola has joined #rubinius
whatasunnyday has joined #rubinius
meh` has quit [Ping timeout: 272 seconds]
meh` has joined #rubinius
whatasunnyday has quit [Quit: whatasunnyday]
whatasunnyday has joined #rubinius
whatasunnyday has quit [Client Quit]
havenwood has quit [Remote host closed the connection]
diegoviola has quit [Remote host closed the connection]
havenwood has joined #rubinius
havenwood has quit [Remote host closed the connection]
havenwood has joined #rubinius
havenwood has quit [Remote host closed the connection]
meh` has quit [Ping timeout: 272 seconds]
josh-k has quit [Remote host closed the connection]
josh-k has joined #rubinius
josh-k has quit [Ping timeout: 250 seconds]
carlosgaldino has quit [Ping timeout: 260 seconds]
carlosgaldino has joined #rubinius
josh-k has joined #rubinius
Bwild has quit [Ping timeout: 272 seconds]
max96at|off is now known as max96at
eregon_ is now known as eregon
meh` has joined #rubinius
<yorickpeterse>
weow, class.cpp is quite the rabbit hole
postmodern has quit [Quit: Leaving]
<yorickpeterse>
ugh, if only Rbx would actually call String#initialize and such from C++, then this object tracer would be super trivial to implement
<brixen>
yorickpeterse: I'll be gentle, your approach sucks :)
<brixen>
so, I'm going to see about removing prebuilt llvm support
josh-k has quit [Remote host closed the connection]
<brixen>
who wants to help package llvm for ancient operating systems
<brixen>
like ubuntu 12.04
josh-k has joined #rubinius
josh-k has quit [Ping timeout: 272 seconds]
havenwood has joined #rubinius
diegoviola has joined #rubinius
<brixen>
we need a more sophisticated panic mechanism than rubinius::bug
havenwood has quit []
havenwood has joined #rubinius
<yorickpeterse>
brixen: hey I never said it was a good approach
<yorickpeterse>
I'm literally just throwing shit around to see what sticks
<yorickpeterse>
(before I come up with something decent)
<brixen>
yeah, I'm trying to save you from being shit-splattered :)
<yorickpeterse>
too late :P
<brixen>
ok, I'm trying to save you from being smothered in shit
<yorickpeterse>
mainly I'm just curious how one would run a hook in Ruby at some point in the VM, in this case whenever _any_ object is allocated
<yorickpeterse>
I suspect I'm doing it before #initialize is called, which breaks things (sometimes), but I'm not sure yet
<brixen>
you wouldn't
<yorickpeterse>
can't even find out again how I figured to put it in Class::allocate()
<brixen>
the thing you need to completely internalize is that running hooks is bullshit
<brixen>
you don't want to do that
<brixen>
you don't want to create that
<yorickpeterse>
well ideally there wouldn't be a need for it, but C++ objects (e.g. Strings allocated in C++) don't go through Ruby
<brixen>
it's the path to total uselessness and pain
<brixen>
they don't need to
<yorickpeterse>
If that were the case, this is all you'd need: class Object; def after_allocate(location); ....; end; end
<yorickpeterse>
(or something like that)
<brixen>
have you read the dtrace paper?
<brixen>
if not, stop what you are doing and go read it
<brixen>
there's a tiny little pdf link on that page
<yorickpeterse>
It's worth mentioning that I'm actually not trying to build something that makes sense, or is even usable. Part of it is to figure out how that pesky VM really works in C++
<yorickpeterse>
So at this stage I don't really care too much about what would be the right way of doing things
<brixen>
"I'm not trying to build a fire here, I'm just seeing which rocks make sparks when I hit them against my head"
<brixen>
carry on then :)
<brixen>
the false dichotomy is that you can both learn and produce something useful
<brixen>
well that sentence kinda sucks
<brixen>
the false dichotomy is that you can't do something useful while exploring and learning
<brixen>
start out with a basis of established knowledge and skill
<yorickpeterse>
I'm perfectly fine with that when it's mainly about learning things
<brixen>
amplify your work many OOMs
<brixen>
you don't want to learn that hitting yourself in the head with rocks hurts
<yorickpeterse>
Having dtrace/systemtap in would be neat, but that's a few steps ahead
<brixen>
I mean, you *may* want to learn that
<yorickpeterse>
I'd still need to figure out where to hook it in for example
<brixen>
we don't want dtrace in, but it will get you thinking about how you really want to build the tool
<brixen>
dtrace still can't deal with the internals of a managed runtime well
<brixen>
and it's not really the problem to solve
<brixen>
but think about how dtrace solved the problem that it was trying to solve
<yorickpeterse>
I don't see us reinventing dtrace any time soon
<brixen>
basically, I'm offering you 2 things: less frustration and more success
<brixen>
whether your success is only learning or building something useful
<yorickpeterse>
granted, being able to do something like this would be really baller: rbxprof --type allocate --group-per-class path/to/some/script.rb
<yorickpeterse>
which then would profile allocations, group them per class, print them, without any code changes required
<yorickpeterse>
(both userland and vmland code)
<brixen>
indeed
<brixen>
you have the right idea :)
<yorickpeterse>
Well that would be my end goal, but the first step is to see where exactly allocations happen (for example)
<yorickpeterse>
and I'm doing that by hammering my head into a wall
<brixen>
we know exactly where they happen
<yorickpeterse>
Well in this case I don't, or not entirely just yet, and it's not really documented
<yorickpeterse>
I bumped into Class::allocate()....somehow, but I can't remember it :/
<brixen>
brb, re-reading the dtrace paper :p
<yorickpeterse>
Reading parts of it atm, as well as systemtamp docs
<yorickpeterse>
curious to see if that would require litering the code with systemtap_emit_whatever("some_event_name") or not
<yorickpeterse>
Since dtrace does require some code instrumentation by the looks of it
<yorickpeterse>
Granted, Ruby would be nicer for this than <insert-custom-profiling-language>
<brixen>
think simpler
<brixen>
don't think about adding dtrace
<brixen>
we already have dtrace probes
<brixen>
they may in fact work kinda ok for allocation, that's not the point
<brixen>
and they plain won't work will for more complicate vm operations
<brixen>
we're not going to expose very Ruby-centric stuff to dtrace C pointer chasing
<brixen>
for example
<brixen>
try to really grasp the design of dtrace and the rationale for that design
<brixen>
s/work will/work well/
<brixen>
wow, my battery is draining like crazy today
|jemc| has joined #rubinius
diegoviola has quit [Quit: WeeChat 1.0.1]
<yorickpeterse>
So assuming the code would not have any explicit event emitting stuff in, how would you go about hooking into a method somewhere half-way its body?
<yorickpeterse>
e.g. typically a profiler can only "wrap around" a method, not hook itself into its inner guts
<yorickpeterse>
(or any other tracer like tool)
<brixen>
what does the inner guts of a method look like?
<brixen>
or better, what is a method?
<yorickpeterse>
In Ruby you could _probably_ inject stuff in the bytecode, but I don't see that happening in C++
<yorickpeterse>
(that is, inject your tracing instructions somewhere in the middle of a method's body)
<yorickpeterse>
assuming that doesn't wreak havoc
<brixen>
indeed, which is a big assumption
<brixen>
but one that dtrace addresses
<yorickpeterse>
tl;dr I want arbitrary tracing without code modification, ideally without event emitting being put in place explicitly, using Ruby
<yorickpeterse>
so basically I want a unicorn
<brixen>
I don't think you want arbitrary tracing
<brixen>
above you showed more specifically what you want
<yorickpeterse>
Well yeah, that's one use case
<yorickpeterse>
but you should also be able to use it just to print what gets called
<brixen>
the general is the enemy of anything useful
<yorickpeterse>
or you could maybe use it as a replacement for the usual crappy, slow profilers
<yorickpeterse>
The interest for allocation tracing comes mainly from Oga, and working on some perf stuff for Rbx
<yorickpeterse>
Right now it's next to impossible to see how many objects method X allocates
<yorickpeterse>
MRI has some disgusting code that stops the GC, but fuck that
<yorickpeterse>
That in turn could seriously skew the results too
<yorickpeterse>
also not thread-safe, but that's not really a problem in MRI anyway
<yorickpeterse>
Hm, seems Systemtap does have some sort of facilities for hooking into a specific line number
<yorickpeterse>
"This makes it unnecessary to maintain a tapset script, and instead involves adding calls to macros from <sys/sdt.h> into your program. These calls are source compatible with dtrace on other platforms, so the same source code compiled on a system that doesn't have systemtap available"
<yorickpeterse>
Hm that would be interesting if it's actually compatible with dtrace
<yorickpeterse>
ha, it actually uses the same dtrace macros
JohnBat26 has joined #rubinius
<brixen>
yorickpeterse: btw, what was your tweet re compiling to asm about?
<brixen>
yorickpeterse: did you use Rubinius::Metrics.data to count the allocations?
<yorickpeterse>
also I totally used a factorial for a benchmark :P
<brixen>
"I know I'm asking a completely ignorant question"
<brixen>
ok...
<yorickpeterse>
The metrics are a nice step, but they hide _what_ was allocated
<yorickpeterse>
That is, you could use it to see how many large objects immix allocated, but not what or what their size was
<yorickpeterse>
or where
<brixen>
I'm surprised there's no comment about using Java (or JRuby) in that thread
<brixen>
yorickpeterse: so, are you trying to profile or optimize at this point?
<brixen>
they are very different activities
<yorickpeterse>
First profiling, then seeing if I could optimize things based on that
<yorickpeterse>
but it's with the idea in mind that I'm trying to get my lexer/parsers to do as little as possible (thus _hopefully_ leading to faster parsing times)
<brixen>
so, if you allocate 500 objects that are 10 bytes or 50 that are 100, etc, what difference would it make?
<yorickpeterse>
In theory, none, but that's why I'd need to know what is allocated, not just if something is allocated
<yorickpeterse>
e.g. if you allocate a bunch of Fixnums, meh those don't have a huge overhead
<yorickpeterse>
but having a bunch of big strings, that's going to hurt if that happens a few million times
<brixen>
Fixnums aren't allocated
<brixen>
if you set up statsd, you could have a ton of information and actually look at correlations
<yorickpeterse>
still have to mess around with that
<brixen>
benchmarks won't tell you systemic information
<brixen>
how the app behaves over time is the only useful information about your app
<brixen>
you *may* be able to find correlations between a benchmark and an app
<brixen>
but that will always be inferior to the apps actual behavior
<yorickpeterse>
I'm aware, but the metrics themselves are not enough in finding and solving a potential bottleneck. The way I see them, and I could be wrong here, is as the initial step
<yorickpeterse>
That is, if the metrics are off, start using more sophisticated tools
<brixen>
yorickpeterse: have you actually compiled the allocation tracking support and tried to use it?
<brixen>
the metrics are never off
<brixen>
unless you mean something else
<yorickpeterse>
What I mean is if they're different from what you expected
<yorickpeterse>
that is, you expected method X to result in metrics Y, but that's not the case
<brixen>
oh, the measurements are off
<brixen>
or better, the measurements are not what you expected
<yorickpeterse>
yes
<yorickpeterse>
Also regarding the allocation tracking, that would only work if you'd have access to the objects
<brixen>
you get a time-series data out of rbx for the number of objects allocated
<yorickpeterse>
it wouldn't work if you profile method X, which calls Y, of which you don't have access to the raw objects that it might allocate
<brixen>
you can correlate that to phases of your app
<yorickpeterse>
oh? I thought it just added location info to objects
<brixen>
the metrics for object allocation are a guage
<brixen>
if you plot that, you can see all kinds of info
<brixen>
rate, avg, blah blah
<brixen>
go set up statsd :p
<yorickpeterse>
wait, I'm talking about the @__allocation_site__ ivar set when allocation tracking is compiled/enabled
<brixen>
yeah, I'm betting you don't need to know precisely what object is allocate where at this point
<brixen>
that's why I asked if you were profiling or optimizing
<brixen>
you have massive profiling info at your finger tips
<brixen>
and you're all like, "meh, I want my head to hurt more"
<yorickpeterse>
haha
<yorickpeterse>
Only sometimes
<yorickpeterse>
Though I don't see how the metrics would allow me to show how many, say, Strings are allocated in method X
<yorickpeterse>
unless I've completely looked over that
<brixen>
granted, setting up graphite is way more work than it should be
<brixen>
we'll need to fix that at some point
<brixen>
they won't show you how many Strings in method X at this point
<brixen>
but you haven't shown that how many Strings in method X matters vs how many objects period
<brixen>
and you probably can't
<brixen>
so you should start by looking at the systemic behavior
<brixen>
both in your benchmark and in a Rails app
<brixen>
or whatever the context is that you're actually using Oga
<chrisseaton>
yorickpeterse: have you read my paper on zero overhead tracing for Ruby? We can add and inline arbitrary code to running compiled methods and then recompile with them there.
<yorickpeterse>
oh wait, that's actually slower than MRI
<chrisseaton>
yorickpeterse: I wonder why the release is after RubyConf - I would have thought they'd want everyone talking about it if it was something special
<yxhuvud>
which might be good enough for certain app types.
<yorickpeterse>
"Bear in mind that those are synthetic benchmarks and real world throughput of real applications should give us different behavior"
<yorickpeterse>
Then why the fuck benchmark hello world
<yorickpeterse>
lets see if we can play benchmark bullshit bingo
<yorickpeterse>
No info about the underlying system/hardware: check
<yorickpeterse>
Some vague benchmark of which they don't say what tool they used: check
<yorickpeterse>
No info about JIT warmups, etc: check
<yorickpeterse>
No info about their SQL benchmarks: check
<yorickpeterse>
No Rbx benchmark: check
<yorickpeterse>
so yes my friends, so far it looks like bs
<Rotonen>
storing partials into memcached with the binary protocol has gone out of fashion, or what?
<yorickpeterse>
clearly we should just compile HTTP requests to assembly
<yorickpeterse>
because then it would be fast
<jnh>
morning
elia has quit [Quit: Computer has gone to sleep.]
havenwood has quit [Remote host closed the connection]
havenwood has joined #rubinius
havenwood has quit [Remote host closed the connection]
<yorickpeterse>
brixen: so I just realized something, dtrace/systemtap is something I've tried to hack in Ruby about a year ago :P
<yorickpeterse>
That is, at some point I needed tracing for ruby-lint and I was like "Fuck, I don't want to add debug statements everywhere. It would be baller if you could define external blocks of Ruby code to be executed when event X happens"
<yorickpeterse>
But it's next to impossible to get a block of Ruby code to run for an arbitrary line number
<yorickpeterse>
Maybe you can do it in MRI using TracePoint, not sure
<yorickpeterse>
but there sure as hell is no cross-implementation way
<chrisseaton>
yorickpeterse: set_trace_func does that - doesn't everyone support that? I know Rubinius has something slightly different but it does the same thing doesn't it?
<yorickpeterse>
MyTracer.when('some_file.rb:123') { ... } # this would be neat
<yorickpeterse>
chrisseaton: we don't have set_trace_func
<chrisseaton>
yorickpeterse: maybe implement that, and then build your trace on top of it
<yorickpeterse>
TracePoint basically makes set_trace_func less shit
<yorickpeterse>
don't think we have the facilities for set_trace_func either
<|jemc|>
I would think something that filters by module and/or method in one or more layers would be more useful than filename/lineno
<yorickpeterse>
|jemc|: you'd be able to do both
<|jemc|>
run this block when MyThing#foo calls OtherThing#bar
<yorickpeterse>
line numbers are useful when you would want a probe in the middle of a method
<|jemc|>
based on Rubinius::Location properties
<|jemc|>
and being able to filter by more than one location on the call stack
<|jemc|>
in case you don't care about any of the other calls to OtherThing#bar
<|jemc|>
I mean, yes you could implement that functionality on top of an underlying facility for filename/lineno, but I think that this is actually more useful and easier and cleaner to implement
<|jemc|>
although I suppose filename/lineno are also properties of Rubinius::Location
<yorickpeterse>
in the mean time my most used snippet will still remain `require 'pry'; binding.pyr` :/
<yorickpeterse>
* pry
<|jemc|>
there's also pry-rescue plus raise "boom" :)
<yorickpeterse>
Oh I'm aware of all the plugins, but they all require code modifications
<yorickpeterse>
maybe if I bug brixen long enough he'll solve that problem too :P
<jnh>
I like to use pry-byebug on MRI, but I haven't looked at how much work it would be to make a version that works with Rubinius::Debugger