gmcabrita has quit [Quit: Connection closed for inactivity]
jemc has quit [Ping timeout: 246 seconds]
amclain has quit [Quit: Leaving]
gmcabrita has joined #ponylang
_whitelogger has joined #ponylang
prose[m] has quit [Remote host closed the connection]
srenatus[m]1 has quit [Remote host closed the connection]
M-hrjet has quit [Remote host closed the connection]
irx[m] has quit [Write error: Connection reset by peer]
buchanon[m] has quit [Remote host closed the connection]
buchanon[m] has joined #ponylang
cousin_luigi has joined #ponylang
<cousin_luigi>
Greetings.
<SeanTAllen>
hello cousin_luigi
<cousin_luigi>
SeanTAllen: I'm struggling with bug #1716 .
<SeanTAllen>
what version of gcc do you have installed?
<cousin_luigi>
SeanTAllen: 6.3
<SeanTAllen>
so i see it can't find `ld`
<cousin_luigi>
SeanTAllen: ld is present in /usr/bin
<SeanTAllen>
is ld.gold installed?
<SeanTAllen>
cousin_luigi: i left some questions on the issue
<cousin_luigi>
SeanTAllen: missing ld.gold seems to be the culprit.
<cousin_luigi>
SeanTAllen: Do you still need the other questions answered?
plietar has quit [Remote host closed the connection]
<SeanTAllen>
did you fix the issue cousin_luigi ?
<SeanTAllen>
ah it appears you did.
<SeanTAllen>
could you do a PR for installing on openSuse that adds info on how to install on openSuse including installing ld.gold?
<cousin_luigi>
SeanTAllen: I'm packaging it for openSUSE.
<SeanTAllen>
ah ok
<SeanTAllen>
well then... when you have it packaged, can you do a PR to add how to use your instructions to the PR?
<cousin_luigi>
SeanTAllen: Right now I'm having another problem: I don't have a network connection during the build, so one of the tests fails. How can I disable it?
<cousin_luigi>
SeanTAllen: Yes, I'm familiar with it.
<staticassert>
My process is getting killed. I think it's eating too much memory. My assumption with the code is it would open the file, read the contents, and then at the end of scope the contents would be GC'd/ dropped. https://gist.github.com/insanitybit/dd94a4ca99bb630d943e84b5bda421a2
<staticassert>
The assumption doesn't necessarily seem true since I'm losing memory super fast.
<cousin_luigi>
TBH I've followed the arch package and it's rather crude to be submitted upstream.
<cousin_luigi>
SeanTAllen: Also, I could use a way to test if things really work as intended once everything is installed from the package. What would you recommend?
<SeanTAllen>
what do you mean "really work as intended"?
<SeanTAllen>
something beyond running the tests or compiling and running a couple of examples?
<SeanTAllen>
homebrew verifies things are good by compiling and running the "hello world" example cousin_luigi
<staticassert>
Nope haha still confused about why my program eats so much memory
<staticassert>
I think maybe I'm just opening a massive file
<SeanTAllen>
staticassert: ill take a look in bit
<staticassert>
Thanks
<SeanTAllen>
on a call with sylvan talking about a talk abstract
<staticassert>
Nice
<staticassert>
I was thinking of talking to the person who runs the local rust meetup and trying to talk about Pony
<staticassert>
I think rust users would be very open to the language
<cousin_luigi>
SeanTAllen: Yes, something like that. Will look for the homebrew script I suppose.
jemc has joined #ponylang
<cousin_luigi>
SeanTAllen: I'm quite new to all this and I must be doing something wrong. How do I run my helloworld program? I must be getting the CLI syntax wrong.
<cousin_luigi>
SeanTAllen: also where can I find the source rpm packages for fedora and the like? I'd like to compare the spec file to what I did.
<staticassert>
cousin_luigi: when you run 'ponyc' it should generate a binary file, which you can execute with ./<filename>
<SeanTAllen>
thank you staticassert
<SeanTAllen>
cousin_luigi: the rpm files are in bintray
<cousin_luigi>
SeanTAllen: But I could find only the binary files, not the source ones.
<SeanTAllen>
what sort of memory usage are you seeing staticassert ?
<cousin_luigi>
staticassert: But where should I put the source for it to be built?
<staticassert>
It shoots up to 50% and then my program dies with 'process killed'
<staticassert>
cousin_luigi: are you trying to build ponyc from source? Or are you trying to use ponyc to build a pony program?
<SeanTAllen>
cousin_luigi: i'm not sure about the rpm source stuff. killerswan handles all that and i haven't looked into it. you could open an issue and ping him on GitHub or send an email to the pony dev list.
<cousin_luigi>
staticassert: Both actually. The first part is done and I'd like to test it's working with the latter.
<cousin_luigi>
SeanTAllen: k, will do that
<SeanTAllen>
if you change directory to examples/helloworld <-- might be wrong
<SeanTAllen>
you run `ponyc` in that source directory
<SeanTAllen>
pony will compile all the files and link them
<SeanTAllen>
then you can run
<staticassert>
So if you have a 'main.pony' in folder /foo/ (so foo/main.pony) you would do /path/to/ponyc ./main.pony
<SeanTAllen>
./helloworld
<SeanTAllen>
the binary will have the name of the directory
<staticassert>
if you are in the /foo directory
<SeanTAllen>
what is 50% of memory staticassert ?
<staticassert>
SeanTAllen: like in htop it climbs to 45-50% before dying. So like 4GB I guess
<staticassert>
The memory thing might be a red herring. It could be another issue entirely.
<staticassert>
'process killed' doesn't give much info though
<cousin_luigi>
ohhh, I understand now
<cousin_luigi>
thanks
<SeanTAllen>
staticassert: my guess without running it is the size of the strings for searcher
<SeanTAllen>
do you have memory issues if Searcher does nothing, but the test of of the program does its thing?
<staticassert>
If I comment out the part where it reads the files there's no problem
<staticassert>
I'm currently having it print the file sizes so I can see if something is huge
<staticassert>
The largest file it scans is 41MB
<staticassert>
There are 10 actors performing the search so at any given them I should have < 400MB of file contents in memory
<SeanTAllen>
do you spawn a new actor for each search even if its limited to only 10 concurrent?
<SeanTAllen>
hmmm
<SeanTAllen>
no
<SeanTAllen>
only 10
<jemc>
SeanTAllen: FYI I fixed the ponylang-main bot issue yesterday, and repaired the broken changelog entries by hand
<SeanTAllen>
jemc: awesome
<SeanTAllen>
staticassert: i've never done file reading like this...
<SeanTAllen>
im assuming yes, but as ive never done it...
<SeanTAllen>
also, i cant get that gist to compile
<staticassert>
It must, yes. The env.out.print that finds "foo" in files is getting printed to the screen.
<staticassert>
huh really?
<staticassert>
That is odd..
<SeanTAllen>
local env issue
<SeanTAllen>
thats better
<SeanTAllen>
it was using ponyc 0.9.0
<staticassert>
Ah
<staticassert>
It may be that in some cases that loop is not terminating, but the worst case scenario is ~400MB of memory usage *if* every actor opens a 40MB file (the largest I have)
<SeanTAllen>
its happily sucking up memory for me
<staticassert>
no crash?
<SeanTAllen>
i killed it eventually
<SeanTAllen>
osx was quite happy to create more compressed memory
<SeanTAllen>
so, i'm triggering a gc after each search run
<SeanTAllen>
still lots of memory being gobbled up
<SeanTAllen>
file.read_string(file.size()) will read the entire file
<SeanTAllen>
i think something wasnt getting an errno to leave your loop
<staticassert>
Hm. That's odd.
<staticassert>
I wonder why the errno wasn't working
<staticassert>
Cool though, it doesn't crash with "../../" but now it does with "../../../" :P so I'm going to check if that's just due to a huge file
<SeanTAllen>
now
<staticassert>
In reality I should just write this so that it doesn't load up every file in memory
<staticassert>
To be clear it crashed with ../../../ before too
<SeanTAllen>
well if you look at how read_string works, you'll see why
<SeanTAllen>
do you want to learn a trick staticassert ?
<staticassert>
sure
<SeanTAllen>
that can help when diagnosing "gc issues"
<SeanTAllen>
@pony_triggergc[None](this)
<SeanTAllen>
that ffi call will set the actor to be gc'd next time it does a check
<SeanTAllen>
which would be after each method call generally
<SeanTAllen>
it can be a nice way to quickly check what happens if gc is running
<SeanTAllen>
its much slower than normal
<SeanTAllen>
because lots of gc
<SeanTAllen>
but it can help
<SeanTAllen>
figure things out sometime
<staticassert>
nice, thanks
<SeanTAllen>
i put that in first to rule out a gc related issue and then started looking at your while that didnt look right to me
<SeanTAllen>
you're welcome
<staticassert>
It would be really cool if you could attach to the pony process and get it info on actors - like 'this actor has n messages in its queue, this actor is uginy n mb memory'
<staticassert>
but that's probably a lot of overhead to support that
<staticassert>
either way I think I just need to rewrite this to do an incremental search - if I do line by line search, which I need anyways, it should solve all of these issues
<staticassert>
can resources leak in pony btw? Like if I have a file that I open, do I need to close it? Or is that handled by RAII/ the GC?
<SeanTAllen>
in theory it should be closed
<staticassert>
Hrm. I have 'for line in FileLines(file) do' - is the 'line' going to be freed at the end of every iteration? The process is still being killed/ taking up tons of memory.
<SeanTAllen>
staticassert: replied to eail
<SeanTAllen>
gc only runs at the end of a behavior
<SeanTAllen>
at the end of each gc run, it wont run until it is using 2x the amount of memory left at the end of this gc run
<SeanTAllen>
so if the actor heap was 2 megs, it wouldnt trigger another gc until at least 4 megs was used
<SeanTAllen>
but if it reset to 2 megs after that gc
<SeanTAllen>
then it would run again at 4 megs
<SeanTAllen>
well, at the end of a behavior once it was using 4 megs
jemc has quit [Ping timeout: 240 seconds]
jemc has joined #ponylang
<staticassert>
SeanTAllen: right, but because 'line' goes out of scope it could be freed without GC
<staticassert>
Basically I want to search the line and then immediately free the memory before moving on to the next line.
<jemc>
staticassert: under some circumstances we're able to convert allocations to stack allocations, in an LLVM optimization pass
<jemc>
but in general your program shouldn't rely on those optimizations to function
<staticassert>
I don't think it would help here unless I were reading in a fixed length into a buffer anyway
<staticassert>
It's more that I need to search through a file, but I don't want to have file_size amounts of memory in use to do so.
<staticassert>
So I'd like to read one line, search the line, free the line.
<jemc>
staticassert: you'll have to break up your loop to be a self-calling behaviour
<jemc>
which means that whatever result you're grabbing will need to be returned to the caller asynchronously, using a callback function or other notifier
<staticassert>
Hm. So the only thing being done with the result is I print something out. Could I hand each line over to another actor, and then just print it in the other actor? Would the memory then be freed for every line?
<staticassert>
Seems like there should be a way to free a value if it's iso, like let _ = consume foo should free the foo memory
<jemc>
we can't guarantee that memory will be freed after every behaviour execution - but memory will *never* be freed by the GC *inside* a behaviour
<jemc>
so you've got to break it up into multiple behaviour executions and trust the GC to "do the right thing" and keep memory from getting too high
<staticassert>
That approach would work, but it seems like it would be easier to just allow an explicit free (not from the GC) for anything that could be consumed
<staticassert>
That way I wouldn't have to code around the GC
<jemc>
thing is that doing this suggested pattern of not having a single massive loop execute in a single behaviour is also good for other reasons
<jemc>
for example, it avoids hogging a scheduler thread for the duration of the loop - each iteration can be interleaved with other behaviour executions, if each iteration is itself a behaviour execution
<jemc>
to put it another way, a behaviour execution is the unit of atomicity in Pony, and doing a huge atomic operation is a bit of an anti-pattern
<staticassert>
that's an interesting way to think about it
<jemc>
breaking it up will aid the GC, but also help your actor to run more smoothly alongside other actors, and share scheduler time better
<staticassert>
yeah, that makes sense
<jemc>
let me know if you need help figuring out how to translate your loop to a self-calling behaviour
<staticassert>
Thanks, I'll let you know. Gotta go feed myself before I start more coding though :P thanks for the help so far
<SeanTAllen>
You're welcome staticassert. Good luck!
jemc has quit [Ping timeout: 260 seconds]
jemc has joined #ponylang
jemc has quit [Ping timeout: 240 seconds]
<staticassert>
Does FileLines(file) load the entire file into memory and then iterate over it?
<SeanTAllen>
if you read the entire file line by line into a method then you will have allocated memory for all of the file
<SeanTAllen>
if you check out line, you can see it starts by allocating enough memory for a line as whatever the size of the last line was
<SeanTAllen>
inside that
<SeanTAllen>
it takes the open file descriptor and reads some data into a string
<SeanTAllen>
each call will allocate new memory
<SeanTAllen>
and have a new string allocated
<SeanTAllen>
can you share your latest code staticassert ?
<staticassert>
The latest code is in that gist
thm has left #ponylang [#ponylang]
<SeanTAllen>
right that usage of line isnt going to help you
<SeanTAllen>
lets talk about 23 to 25 for a moment
<SeanTAllen>
line 23 is reading the file in one line at a time
<SeanTAllen>
allocating in the same behavior, memory for each line
<SeanTAllen>
lots of little chunks
<SeanTAllen>
if it were stack allocated, it could be freed
<SeanTAllen>
possibly
<SeanTAllen>
but handing it to line printer means that cant possibly happen anyway
<SeanTAllen>
it needs to be on the heap of the Searcher actor
<SeanTAllen>
and will remain there until its gc'd
<SeanTAllen>
which at the earliest happens when we leave this behavior
<staticassert>
I see, so they're all being held in memory for the entire search through the file
<SeanTAllen>
consuming a variable does not cause it to be copied and transfered to another actor's heap
<SeanTAllen>
ya
<staticassert>
Is there a way to do this without relying on the optimizer to stack allocate? If I could just explicitly free each line that seems like a solution.
<staticassert>
or if the line simply lived till the end of the block
<SeanTAllen>
you could drop down to c/ffi to do things but then you'd need to write your own file handling as well
<SeanTAllen>
are you hitting a really large file or something that takes up a ton of memory when you read it in?
<SeanTAllen>
when jemc was talking about breaking up the behavior calls, i think he meant something like
<SeanTAllen>
which you can see in TCPConnection where when its doing a read, it read up to X amount of data and then calls pending_read to give up the scheduler (which also allows a gc to happen)
<staticassert>
Yeah, that could work.
<SeanTAllen>
and if you only want to say "hey its in there"
<staticassert>
They have some incredibly interesting features, specifically around debugging.
<ericbmerritt_>
just out of curiosity, it looks like llvm will have an avr backend in v4. I wonder if it will be possible then to use pony on arduino?
prettyvanilla has joined #ponylang
<plietar>
ericbmerritt_: I'm not sure Pony on arduino would be a good fit
<plietar>
Arduino is really constrained in terms of resources
<ericbmerritt_>
this is true, I like the idea of playing around with arduino, hate the idea of doing it in an imperative language
<plietar>
An arduino due is probably an easier target for pony
<plietar>
Can pony even do bare metal at all at the moment ?
staticassert has quit [Quit: Page closed]
Praetonus has joined #ponylang
<Praetonus>
ericbmeritt_ plietar: The pony runtime currently requires an OS, for threads and things like that. I don't think pony on arduino will happen
plietar has quit [Remote host closed the connection]
plietar has joined #ponylang
plietar has quit [Remote host closed the connection]
Praetonus has quit [Quit: Leaving]
plietar has joined #ponylang
plietar has quit [Remote host closed the connection]