<cess11_>
Logged off a little more than an hour ago.
<freeemint__>
ok
Regenaxer has joined #picolisp
<Regenaxer>
freeemint__, sorry! Was offline - Installed a new router.
<freeemint__>
No problem
<freeemint__>
I found the problem
<Regenaxer>
cool!!!
<Regenaxer>
What was it?
<freeemint__>
yout got three options: either i can point you to a reference implementation (which makes the problem very obvious), or i can tell you what is wrong on an abstract level and how i think it should work, or we can play a little question and answer game which might expose your wrong assumptions
<freeemint__>
1, 2 oder 3 ?
<Regenaxer>
3
<freeemint__>
Can you describe what your code should produce in terms of packets?
<freeemint__>
(what protocol they have, what the content is, how many there are)
<Regenaxer>
Which kind of packets?
<Regenaxer>
not TCP, right?
<Regenaxer>
The messages?
<Regenaxer>
That was my question yesterday, how they are structured
<Regenaxer>
eg. header each time
<freeemint__>
Yeah, then i take another approach, : What happens with packets which are not wanted?
<Regenaxer>
What do you mean with "packet"? Messages?
<Regenaxer>
"abc" etc
<freeemint__>
i mean packets
<Regenaxer>
TCP?
<freeemint__>
Yes
<Regenaxer>
This is another layer
<Regenaxer>
Not handled here
<Regenaxer>
Socket connection
<freeemint__>
that is where you are wrong
<Regenaxer>
ok, then tell me
<freeemint__>
Can you still answer my question first
<Regenaxer>
I have no control over packets
<Regenaxer>
I have a connection
<Regenaxer>
stream of bytes
<Regenaxer>
There is no concept of packets on the pil level
<freeemint__>
Just assume that i know what i am asking of you
<Regenaxer>
A socket option?
<freeemint__>
What happens when you sent package (cotaining a http header) to a client but the client is not waiting for it
<cess11_>
Wanted by whom? Usually they are dropped.
<freeemint__>
cess11_: That is right they are dropped
<Regenaxer>
I don't "send a packet" anywhere
<freeemint__>
Regenaxer yes
<Regenaxer>
A TCP connection is opaque
<Regenaxer>
stream of bytes
<freeemint__>
I tell you what a real sse is.
<Regenaxer>
ok
<freeemint__>
a real sse is one tcp stream which starts with an http header and is kept alive by the server
<cess11_>
Haven't had time to take a more thorough look at SSE:s yet, did however set up a new gate and pil app this morning.
<freeemint__>
demo.howopensource.com/sse/ for the app i think
<freeemint__>
they also have a source doe
<freeemint__>
but php
<freeemint__>
Is it possible to have a function print in to two output streams?
<freeemint__>
(assuming functions is not deterministic so recalling doe snot count)
<Regenaxer>
no, there is always only one stream at a moment
<freeemint__>
I mean in general
<freeemint__>
somehow
<Regenaxer>
(out '(tee ...) ...
<Regenaxer>
Generally in Unix? Also only one file descriptor for one write() operation
<freeemint__>
I am trying to build and tcp shell ... (a thing that (executes picolisp and (shows me the results and output) and sends the output to the server) and (reads incomming messages from the server and outputs them to me))
<Regenaxer>
A Repl?
<freeemint__>
Sorta
<Regenaxer>
or more a terminal
<Regenaxer>
bin/psh does something like that
<freeemint__>
but the output of eval has to go back to me and to the server
<freeemint__>
output as in 'prinl
<freeemint__>
not as in "return value"
<freeemint__>
I got an ugly idea.
<freeemint__>
I could write stuff in to a file and read it out two times
<Regenaxer>
Either with 'tee' as above
<freeemint__>
What is 'tee
<freeemint__>
i find no docs
<Regenaxer>
or (let Val (foo) (out Here ) (out There))
<Regenaxer>
Unix Urgestein
<Regenaxer>
man tee
<freeemint__>
it is not about the return value but about what is printed
<Regenaxer>
yes, the stdout stream kind of
<Regenaxer>
The 'repl' function in @lib/form.l does that too I think
<Regenaxer>
It writes to a temp file first
<cess11_>
The pil udp function eats PLIO, set it up with socat or httpGate and you're almost there.
<Regenaxer>
(out (pack "+" (: home repl))
<Regenaxer>
yep
<Regenaxer>
OK, demo.howopensource.com/sse/stocks.php works
<Regenaxer>
Can you capture the stream with wireshark?
<cess11_>
'(prinl (pack "data: " L "\n\n") would be my next test unless you've already checked.
<Regenaxer>
'prinl' :)
<Regenaxer>
yes, did all that yesterday
<cess11_>
Right.
<Regenaxer>
also "\r\n\n" etc.
<Regenaxer>
also "\r\n\r\n" I mean
<Regenaxer>
The thing is that it *does* work, but only *one* time
<cess11_>
And ht:Out does flush?
<Regenaxer>
funny
<Regenaxer>
yes
<freeemint__>
doesn't ht:out already do (char 13) (char 10)
<Regenaxer>
But it stops before the end of ht:Out is reached
<Regenaxer>
Only "abc" is processed
<Regenaxer>
then the JS EventSource stops
<Regenaxer>
Drives me nuts!
<freeemint__>
i won't blame it on java ... curl does finish early too
<Regenaxer>
thanks for the files!
<Regenaxer>
ok, good to know
<freeemint__>
that is my point
<freeemint__>
curl has the same struggles as js after the message it thinks the stream is closed ...
<Regenaxer>
ok
<Regenaxer>
but it definitely is not
<Regenaxer>
(for L '("abc" "def" "ghi") ... no close
<freeemint__>
Regenaxer: the version of the dump i sent you is from yesterday when we moved the header out of the loop
<Regenaxer>
ok
<cess11_>
The stuff after 'data: ', did you try enclosing it in {}, like {' L '} or something like that?
<Regenaxer>
This is for JSon
<Regenaxer>
should not matter here
<freeemint__>
cess11_: Json is not necessary
<cess11_>
It shouldn't be, but this is the browser, it is a mess.
<freeemint__>
it is not
<Regenaxer>
true
<freeemint__>
the problem is not the browser
<freeemint__>
unless curl is faulty too
<Regenaxer>
cause of curl?
<freeemint__>
which i doubt
<Regenaxer>
T
<cess11_>
I'm not saying anything about what is the problem, I'm assuming weird things might be needed to have the browser or other web clients do the right thing.
<freeemint__>
try to get working on curl the it will work on browser too
<Regenaxer>
cess11_, yes, questioning it from all sides
<cess11_>
Perhaps. curl is quite different from regular browsers.
<Regenaxer>
There is some stupid assumption I made probably
<freeemint__>
Regenaxer: Can you dump what the pil server is sending? (like change the output from a socket to a file?)
<cess11_>
Is that Zappel? I've modified that one quite a bit.
<Regenaxer>
canvas does its own refresh
<Regenaxer>
yeah :)
<Regenaxer>
I have funny apps meanwhile, also with Gis data and OSM
<Regenaxer>
interactive firemen :)
<cess11_>
It is really impressive and nice to work with, I'll be using it to show real time data on a system and some RSS feeds it will listen to later this year.
<Regenaxer>
cool
<cess11_>
The OSM-app is fun too, I learned quite a bit from studying that ER-model.
<cess11_>
I'm pondering using it for showing election statistics, not sure yet though.
<Regenaxer>
Meanwhile there is also @lib/gis.l and @lib/gis.js
<Regenaxer>
much better than the above demo
<Regenaxer>
I can give examples later
<Regenaxer>
You can move objects with the mouse, and they are updated in the DB
<cess11_>
Insane. I'd love some examples when you find the time.
<Regenaxer>
OK, getting close. I'm really excited now to see if sse works
<Regenaxer>
yes, sure
freeemint__ has quit [Ping timeout: 260 seconds]
freeemint_ has joined #picolisp
<Regenaxer>
oh!
<Regenaxer>
Still does not work!
<freeemint_>
(i am switched to something else
<Regenaxer>
ok
<freeemint_>
will spend some time on it later
<Regenaxer>
Thanks for all the help!
<Regenaxer>
my problem now
<Regenaxer>
perhaps wrong test
<freeemint_>
have you got it working for curl?
<Regenaxer>
no, never tried
<freeemint_>
i think it should be the metric of success
<freeemint_>
you could even automate it
<freeemint_>
but use curl -s stream_link
<Regenaxer>
this is not the problem now
<Regenaxer>
I have small chunks now
<Regenaxer>
still only abc
<freeemint_>
throw curl at the stream !
<Regenaxer>
why?
<Regenaxer>
and how?
<Regenaxer>
The chat test will not work
<Regenaxer>
needs another dedicated test
<freeemint_>
ah
<Regenaxer>
with a
<Regenaxer>
###Delay
<Regenaxer>
did you insert this, or overwrite empty lines?
<Regenaxer>
Seems I have one newline too much now
<freeemint_>
look at the lower once they are untouched
<Regenaxer>
count, data, count, data ... instead of single count
<freeemint_>
What means: 25663 = 50749 8002154871166301~
<Regenaxer>
ah, ignore, this is port and sessionid which is now logged for 'psh'
<Regenaxer>
in all GUI programs
<Regenaxer>
Sad
<Regenaxer>
so this was *not* the reason?
freeemint__ has joined #picolisp
freeemint_ has quit [Ping timeout: 260 seconds]
<Regenaxer>
Also, 0 for flg is not good
<Regenaxer>
it is also in HTTP/1.0
<Regenaxer>
Better use 'T'
<Regenaxer>
but that's cosmetics
<Regenaxer>
problem is that it did not help :(
<freeemint__>
I found a potential problem
<Regenaxer>
yes?
<freeemint__>
When you (server 8080 "!start") and (de start () (httpHeader) (content 1) (wait 2000) (content 2)) does not sent as "(content 1) " (waiting 2000 milisecs) "(content 2)" but as (waiting 2000 milisecs) "(content 1) " "(content 2)"
<freeemint__>
it is not relvant to the problem but will become a huge problem later on
<Regenaxer>
Can't be
<Regenaxer>
How do you measure?
<Regenaxer>
you can't measure this way
<Regenaxer>
The stream is buffered
<freeemint__>
look at curl
<freeemint__>
wait ....
<Regenaxer>
Try a (flush) before the wait
<freeemint__>
might make a mistake
<Regenaxer>
I think this is OK
<Regenaxer>
no problem
<Regenaxer>
The output stream is buffered of course
<freeemint__>
"(content 1) " (waiting 2000 milisecs) "(content 2)" You really want that if you are doing streams
<Regenaxer>
yes
<Regenaxer>
(flush)
<Regenaxer>
!
<Regenaxer>
But why would you want that?
<freeemint__>
oh right it works with flushing
<Regenaxer>
More important: Why does SSE not work? Back at the start
<Regenaxer>
I need it tomorrow :)
<freeemint__>
no pressure
<Regenaxer>
Building a chatbot for IBM Watson people
<Regenaxer>
I have a telcon tomorrow and need a strategy
<freeemint__>
and you need to use sse?
<freeemint__>
or you choose to?
<Regenaxer>
I want to
<Regenaxer>
Better than WebSocks
<Regenaxer>
better == simpler :)
<freeemint__>
can you give me an html where i can enter an url and it tries to sse that link?
<freeemint__>
(to test the js site
<Regenaxer>
which js site?
<freeemint__>
js side
<Regenaxer>
of what?
<Regenaxer>
chat?
<freeemint__>
Can you give me an html and a js. which has a text box and a chatfield when i enter an url in to text field and click ok the ESS js client connects to the url in the text box
<Regenaxer>
as yesterday
<freeemint__>
i want to supply my own url not use your server
<Regenaxer>
hmm, I did not release the new pil
<Regenaxer>
with ht:Out changed
<freeemint__>
i got something working which is accepted by the python client and i wanted test it again the JS client
<freeemint__>
(and i know no js)
<Regenaxer>
What did you change? Can I do it here?
<freeemint__>
i am using hard coded stuff
<freeemint__>
no
<freeemint__>
i pastbin the source and you understand
<freeemint__>
Could you test the js client against that
<Regenaxer>
A lot more newlines?
<freeemint__>
Nope
<freeemint__>
i used your c file
<freeemint__>
but i sent your C file over a raw socket and it works
<Regenaxer>
Which C file?
<freeemint__>
the file named "c" you gave me
<Regenaxer>
ah
<freeemint__>
c.tgz
<freeemint__>
un taz.gztet
<Regenaxer>
the output
<freeemint__>
yes
<freeemint__>
if you put the output over a raw socket it works
<Regenaxer>
What do you mean with raw socket?
<freeemint__>
look at the source
<freeemint__>
luke ;D
<Regenaxer>
It is a raw socket to the browser
<Regenaxer>
What is the critical difference?
<Regenaxer>
The wait?
<freeemint__>
I have on idea what is the difference but it works for me
<freeemint__>
sending you file over email again
<freeemint__>
just execute the file and run (go) in the repl
<Regenaxer>
and then? browser?
<freeemint__>
it just a stream
<freeemint__>
but the stream is accepted by the python ess client
<freeemint__>
and curl makes no problem either
<Regenaxer>
I don't understand
<Regenaxer>
how to use it and to see what then
<freeemint__>
can you point a JS ess client to localhost:8080
<Regenaxer>
It must work in the browser
<Regenaxer>
Sorry, interrupt
<Regenaxer>
family issues
<Regenaxer>
need a break
<Regenaxer>
Back later!
<freeemint__>
I want you to test whether it works in the browser.
<Regenaxer>
ok, I'm back later. Sorry
<Regenaxer>
I will meditate meanwhile
<freeemint__>
I wrote a sse server only. (might not be a bad idea in general) which is accepted by the python client but i have not tested it against JS client.
<freeemint__>
i will ramble on
<freeemint__>
my past version was very flawed
<freeemint__>
ignore it for now
<freeemint__>
Regenaxer: How do i set the mimetye?
<cess11_>
In the header.
<cess11_>
Content-Type or somesuch.
<Regenaxer>
ret
<Regenaxer>
httpEcho is best
<Regenaxer>
I think I must debug from the start again to see where it goes wrong
<Regenaxer>
The small chunks was definitely a good idea, and probably necessary
<Regenaxer>
Must be some minor glitch
<Regenaxer>
Ha! As I thought!
<Regenaxer>
Different now
<Regenaxer>
I inserted a 'lisp' line in JS
<Regenaxer>
function serverSentEvent(fun, fld) {
<Regenaxer>
(new EventSource(SesId + "!" + fun)).onmessage = function(event) {
<Regenaxer>
Now all three abc, def and ghi arrive!
<Regenaxer>
This was not the case before the change of chunked!
<Regenaxer>
So now the question is only why it does not appear in the text area
<freeemint__>
Regenaxer: I got it working
<Regenaxer>
good
<freeemint__>
also contentype *needs* to be set to text/eventstream
<freeemint__>
*event-stream
<Regenaxer>
This we have all the time
<freeemint__>
should i send you the zip?
<Regenaxer>
since my first version
<Regenaxer>
no
<freeemint__>
It is an onholy
<Regenaxer>
The problem is different now
<freeemint__>
*unholy abomination
<freeemint__>
It is?
<Regenaxer>
As I just said, the messages arrive now
<freeemint__>
I got working in chrome and firefox
<Regenaxer>
see a few lines up
<Regenaxer>
The 'chat' script?
<freeemint__>
no
<freeemint__>
my own abomination
<freeemint__>
manualy alligned hard coded stuff in my own test bench
<Regenaxer>
ok, good, but it will not help me, right?
<Regenaxer>
cool!
<freeemint__>
So the problem is in JS?
<Regenaxer>
My post() perhaps?
<Regenaxer>
We are getting close
<freeemint__>
Is the problem in JS?
<Regenaxer>
as I said, the chunked thing was important I think
<Regenaxer>
not sure
<Regenaxer>
the JS doesn't do much
<Regenaxer>
let me debug more
<freeemint__>
ok
<Regenaxer>
The big progress is that the above JS *does* called 3 times now
<Regenaxer>
as you remember, the browser called it only once yesterday
<freeemint__>
yeah
<Regenaxer>
This was probably because the chunk was closed with zero
<Regenaxer>
I understand it now
<freeemint__>
good
<Regenaxer>
I debug again between JS and Pil
<freeemint__>
how do i convert a number to hex?
<Regenaxer>
(hex ..)
<freeemint__>
how does print convert everything in to prinable string?
<freeemint__>
(or is there a function which converts sexpr in strings which are the same what print would produce)
<Regenaxer>
yes, 'str'
<freeemint__>
good
<Regenaxer>
Now it works. Seems I must put a delay between the messages
<Regenaxer>
I faintly remember that I read this somewhere yesterday
<Regenaxer>
but cannot find it any more
<Regenaxer>
not sure
<Regenaxer>
And not beautiful, but I can live with it for the moment
<Regenaxer>
No
<Regenaxer>
It is not right
<Regenaxer>
But the problem is in the interaction of JS with the GUI
<Regenaxer>
A home-grown problem, need to dig into it
rob_w has joined #picolisp
<freeemint__>
I am glad i could be of some use
<freeemint__>
Regenaxer: You do not need to put a time delay between messages, just a CR LB delay
<freeemint__>
(my experience
<freeemint__>
Regenaxer: I start the server in the repl manually. then is
<freeemint__>
*I connect my client, then STRG + C in the repl and then i 'bye in the repl and open nano in the shell where the server run before and the browser client is still getting new values.
<freeemint__>
Why
<freeemint__>
Only when i restart the server "endet der Spuk" . (iam running on port 80 with sudo)
<freeemint__>
*restart the client
<cess11_>
Any pil processes in 'sudo ps -a'?
<freeemint__>
(there are still version of client running which do not exist on disk anymore)
<freeemint__>
cess11_: Ôfc ourse
<cess11_>
One of those or something in 'lsof -i :80', then.
<freeemint__>
yes
<freeemint__>
and the number ofcourse matches the number of clients which recieve ghost data
<freeemint__>
i want to know why they are still around?
rob_w has quit [Read error: Connection reset by peer]
<freeemint__>
Does somebody here work for HAGL Biogas oder Mibtec.de ?
<cess11_>
Because you didn't kill them and they still had open connections, I'd guess.
<freeemint__>
I guess so too ... but that is kinda scary ...
<cess11_>
'killall pil'.
<freeemint__>
cess11_: That is not the scary part
<freeemint__>
the scary part is: If you have a security flaw in your server and you fix him all old open connections remain attackable
<freeemint__>
or the scary part is: You can not run two servers on the same machine/user if you want to make sure all old connections are closed
<cess11_>
No. You take them down if you need to.
<cess11_>
Then you lock down iptables, pf, whatever.
<cess11_>
And usually it is nice that one can just replace the app entry point and logged in users doesn't notice.
<cess11_>
If you want scary, kick up the udp backdoor oneliner and start sending it '(call ... expressions..