<Regenaxer>
beneroth, thank you! Good idea with the Reply-To
<Regenaxer>
it is not even IPC
<beneroth>
are the pil IPC pipes fully ok in the children?
<beneroth>
only stdin etc messed up?
<Regenaxer>
it is just reading
<beneroth>
hm.. then maybe how stdin is opened (owning vs sharing access flags etc) ?
<Regenaxer>
yes, kind of
<beneroth>
must be something in the kernel call if it differs between different linux kernels
<Regenaxer>
it depends on the numbers of "a"s
<beneroth>
and/or memory protection
<beneroth>
what
<beneroth>
really
<Regenaxer>
the (fork) must be on a 512 byte boundary
<beneroth>
the comment has an effect? I seriously thought you just had it there for theatrical effect
<beneroth>
omg. a heisenbug.
<Regenaxer>
yes
<Regenaxer>
not heisenbug really
<beneroth>
well in real code yes, until you find the boundary
<Regenaxer>
strange because BUFSIZ is 8192
<beneroth>
not a race condition, T
<beneroth>
hm.. I just wonder.. is there even a way for pil user code to trace garbage collector calls, including gc calls which were issued by memory allocation? (for given lisp code, how many times is the gc implicitly called if the code doesn't contain any explicit (gc) calls)
<Regenaxer>
I did that sometimes
<Regenaxer>
but not user code
<Regenaxer>
Modified the asm sources
<beneroth>
ok. so ad-hoc pil asm debug code?
<beneroth>
ok
<Regenaxer>
tankfeeder, so you could definitely reproduce on pil32
<beneroth>
the aaaa comment is not translated. it is only read into ram and skipped by the reader, no?
<Regenaxer>
This surprises me a lot
<Regenaxer>
beneroth, yes
<beneroth>
is it even in the ram?
<Regenaxer>
it causes the reading to continue later in the stream
<tankfeeder>
archlinux pil32 works
<Regenaxer>
should be in RAM
<tankfeeder>
trying centos
<beneroth>
sorry I only have a rough understanding of that low level. just shooting randomly in case I spark an idea for you. tell me when I'm not helping.
<Regenaxer>
It seems not a bug in Pil, but this usage is illegal? Makes no sense to me yet
<Regenaxer>
(and (fork) (fork) (bye)) is in the parent
<Regenaxer>
The first child continues correctly, with (msg '<1>)
<Regenaxer>
But the second does *not*, it goes somewhere to the wrong point in the input stream
<Regenaxer>
It seems that the file position is modified by the first child
<Regenaxer>
globally
<tankfeeder>
According to "Advanced Programming in the Unix Environment", by Stephens and Rago, the second fork is more a recommendation, and it is done to guarantee that the daemon does not acquire a controlling terminal on System V-based systems.
<tankfeeder>
from stackoverflow
<tankfeeder>
:)
<Regenaxer>
This example is boiled down from a more complicated code, with a loop
<Regenaxer>
no, this is child of a child
<Regenaxer>
httpGate does this
<Regenaxer>
to detach
<Regenaxer>
but here we have 2 children from the same parent
<Regenaxer>
The real code where I hit upon this bug was:
<Regenaxer>
(in *Sock
<Regenaxer>
no
<Regenaxer>
(setq *Port (port 2201))
<Regenaxer>
<Regenaxer>
(loop
<Regenaxer>
(setq *Sock (listen *Port))
<Regenaxer>
(NIL (fork) (close *Port))
<Regenaxer>
(close *Sock) )
<Regenaxer>
<Regenaxer>
(in *Sock
<Regenaxer>
(let? L (rd)
<Regenaxer>
...
<Regenaxer>
It appeared when I inserted new code before the (setq *Port ...
<Regenaxer>
All children except the first did not correctly read the (in *Sock
<Regenaxer>
Took me some time to boil it down to the simple example with two (fork)s
libertas has quit [Ping timeout: 255 seconds]
<Regenaxer>
hmm
<Regenaxer>
It *is* a race condition!
<Regenaxer>
If I wait a little between the forks, it works!
<Regenaxer>
Hmm, got it! What I did was indeed illegal. I checked in Stevens, page 60 and 190
<Regenaxer>
While each file has indeed its own file table entry with the current offset, this is *inherited* to child processes
<tankfeeder>
indeed
<Regenaxer>
Issue closed, I'll avoid such code in the future
<Regenaxer>
Learned something new :)
<Regenaxer>
or - stupid that I did not know :(
<Regenaxer>
This means that one has to be extremely careful with lseek() followed by I/O, a child or parent may run and make a race condition
<Regenaxer>
Pil DB-I/O uses pread() which is atomic though
<Regenaxer>
cc pread(((DbFile)) Z C E)
<Regenaxer>
cc pwrite(((DbFile)) Z C E)
<clacke[m]>
> It takes a very special kind of chutzpah systematically to assault voters, and drag them from polling booths by their hair, and then say that a low turnout invalidates the vote.
<clacke[m]>
of course, he says it means there is no rule of law and no democracy. but this is rule of law. the constitution says Spain is indivisible. it is democracy. the constitution was confirmed by a massive 90% of Catalan voters back in 1978.
<clacke[m]>
what it is not, is anarchy and individual freedom
<beneroth>
Regenaxer, so what did you change?
<Regenaxer>
Nothing at all
<Regenaxer>
Just my script
<beneroth>
so you say pil user code which does (fork) (fork) after each other is illegal?
<beneroth>
I don't follow. is the "file table entry with the current offset" the current position in the input stream, or the current (file-) size of the input stream?
<beneroth>
go on
<Regenaxer>
(loop
<Regenaxer>
(setq *Sock (listen *Port))
<Regenaxer>
(unless (fork)
<Regenaxer>
(close *Port)
<Regenaxer>
... )
<Regenaxer>
(bye) )
<Regenaxer>
(close *Sock) )
<Regenaxer>
So the children do *not* read further
<Regenaxer>
yes, the current offset is in the file table entry
<Regenaxer>
in the kernel
<Regenaxer>
each process has it privately
<Regenaxer>
BUT:
<Regenaxer>
It is inherited to children!!
<beneroth>
so they start with the initial values with were the last values in the parent just before the (fork), right?
<tankfeeder>
why works in linux ?
<Regenaxer>
I was not completely aware of that, though I read it in Stevens years ago
<beneroth>
I still don't get it
<Regenaxer>
It never works
<beneroth>
oh so you never tested on linux?
<beneroth>
that script
<Regenaxer>
just by chance if the child continues reading at the right place
<beneroth>
a, race condition
<Regenaxer>
We *all* tested on linux
<beneroth>
so it IS a heisenbug
<Regenaxer>
no race condition
<Regenaxer>
no
<Regenaxer>
it is buffer arithmetic
<beneroth>
ok
<Regenaxer>
the parent and child *share* the offset
<beneroth>
ah, not copy, but share!
<Regenaxer>
the first child inherits it, and continues reading
<Regenaxer>
but then the second child continues from *that* place
<beneroth>
and the second child gets the values which were already altered by the first child?
<Regenaxer>
Where that is depends on buffer sizes
<beneroth>
ah!
<beneroth>
I see
<Regenaxer>
So it depends a lot on buffer boundaries
<beneroth>
so in practice: the second child might skip some input (or not) depending on stdin read buffering
<Regenaxer>
each process reads a block into its local buffer
<beneroth>
is that the symptom?
<Regenaxer>
but the offset is global
<Regenaxer>
Yes: the second child might skip some input
<beneroth>
so this is the piece I sent with my first link, the one about the file description within kernel space (not part of the file pointer)
<Regenaxer>
if the first read in all its buffer
<Regenaxer>
so that a new read() was necessary
<Regenaxer>
almost, though it was about something different iirc
<Regenaxer>
The point is that processes have private file entries
<Regenaxer>
but not completely private
<Regenaxer>
because inherited
<beneroth>
I would have expected copy-on-write sharing (changing the offset being a write)
<Regenaxer>
yes, me too
<Regenaxer>
this was the error
<beneroth>
ok
<Regenaxer>
but this file entry is in the kernel
<beneroth>
I still don't grok it fully. what is wrong the second pil code?
<Regenaxer>
Works, cause it all fits into a single block
<beneroth>
ah, because it is a short script
<Regenaxer>
Even longer I think
<Regenaxer>
T
<beneroth>
how tricky
<Regenaxer>
yeah, I'm surprised. Never thought about that
<beneroth>
and that is the same for pil32 and pil64, as it uses the same kernel calls which (ofc) works the same internally. right?
<Regenaxer>
I remove it from the distro next time
<Regenaxer>
Exactly
<Regenaxer>
or same buffer size
<Regenaxer>
libc
<beneroth>
I have a similar echo I made once myself, for http
<Regenaxer>
I think the kernel does not matter
<Regenaxer>
oh :)
<beneroth>
(default P 8023)
<beneroth>
(in S (echo))
<beneroth>
(de http (S)
<beneroth>
(close S) )
<beneroth>
(setq P (port P))
<beneroth>
(gc)
<beneroth>
(loop
<beneroth>
(setq *Sock (listen P))
<beneroth>
(NIL (fork) (close P))
<beneroth>
(close *Sock) )
<beneroth>
(task *Sock (http @))
<beneroth>
(http *Sock)
<beneroth>
(bye)
<beneroth>
so this is also prone to this behaviour?
<Regenaxer>
yes, same situation
<Regenaxer>
but it is short enough ;)
<beneroth>
I did this without knowing @misc/reverse.l, independently by my understanding of (fork)
<beneroth>
so this is likely to happen to other people, too
<Regenaxer>
right
<beneroth>
so there should some sort of guidelines into the reference for (fork). what is then the guiding rule to mitigate this error?
<Regenaxer>
Not many dare to work on that level though
<Regenaxer>
Would be good to explain somewhere
<beneroth>
well T, but on other hand that picolisp makes this level so very easy and accessible that it is really really useful for little debug toolings.
<Regenaxer>
But I suspect most people don't understan/ (fork) at all
<beneroth>
probably true. but i found the pil reference pretty easy to understand, once you wrap your head around the fact that you still have the same code stream and just know by the return value of (fork) if you are in the parent or the sibling
<Regenaxer>
yeah
<Regenaxer>
I will fix misc/reverse.l instead of removing it
<beneroth>
*I* still haven't understood fully how to prevent this bug. so the rule is "do not continue with code on top-level for the child process" ?
<Regenaxer>
Perhaps a Wiki article about that would be good
<beneroth>
much better idea!
<beneroth>
both good ideas!
<Regenaxer>
Well, the rule is hard to define. Children should not read an open stream independently?
<Regenaxer>
The file position is not well-defined then
<Regenaxer>
It is indeed hairy
<beneroth>
my main server function (this is an shortened version with only the parts relevant to this discussion) looks like this:
<beneroth>
(de server (Port Func)
<beneroth>
(setq P (port P))
<beneroth>
(loop
<beneroth>
(setq *Sock (listen P))
<beneroth>
(NIL (fork) (close P))
<beneroth>
(close *Sock) )
<beneroth>
(Func *Sock)
<beneroth>
(bye) )
<beneroth>
so is this one fine or not?
<Regenaxer>
This is all right
<Regenaxer>
The whole sexpr is read already
<beneroth>
which one sexpr
<Regenaxer>
The problem is only reading from a file
<beneroth>
ah
<beneroth>
I see
<Regenaxer>
the whole (de ...)
<beneroth>
no read from stdin necessary
<Regenaxer>
T
<Regenaxer>
The test above was also not stdin
<beneroth>
this makes me think the rule I proposed sounds right: Do not do any top-level stuff anymore after (fork) executed
<Regenaxer>
it was a normally opened file
<Regenaxer>
not top-level I would say
<beneroth>
so the problem would apply in the above code if the parent would read some parts of *Sock ?
<Regenaxer>
but file operations
<Regenaxer>
yes, probably
<beneroth>
argh this is extremely hairy
<Regenaxer>
note that it is *not* the parent
<Regenaxer>
it is between the children
<beneroth>
destroy the whole beautiful simplicity of pil (fork)
<beneroth>
ok
<Regenaxer>
Say you open a file
<Regenaxer>
than make a few children which read
<Regenaxer>
then the results are unpredictable
<beneroth>
so the problem is when a input stream (stdin, a common used file, a socket) is read by multiple, closely after each other instanced children ?
<Regenaxer>
if the children read from time to time
<Regenaxer>
Can also be a long time between
<beneroth>
sounds to me as if this does pretty directly apply to the DB files?
<Regenaxer>
The point is that each child may modify the file position
<beneroth>
why isn't it a problem with the db files?
<Regenaxer>
really sick
<Regenaxer>
pread() and pwrite()
<Regenaxer>
atomic
<beneroth>
ah, unbuffred
<Regenaxer>
lseek() + read()
<Regenaxer>
It is buffered
<Regenaxer>
the file offset is still global
<Regenaxer>
but everybody sets it before I/O
<Regenaxer>
so no problem here
<beneroth>
argh ok
<Regenaxer>
This is a strange feature in Unix
<Regenaxer>
but probably necessary
<beneroth>
so (outside of picolisp) people have the same problems but few enough people work on that level and they take care of it?
<Regenaxer>
because open files are inherited
<beneroth>
else there should be usable solution
<Regenaxer>
yes
<beneroth>
ok
<beneroth>
what does then the value "offset" stand for? not the reading position, as this could be different for every process?
<Regenaxer>
It is the reading and writing position
<Regenaxer>
you can modify it with lseek()
<beneroth>
ok. and depending on which io calls the process makes, it re-uses that number or resets it before doing read/write ?
<Regenaxer>
read() starts from that place, and increments it by the amount of bytes
<Regenaxer>
write() too
<beneroth>
pread() and pwrite() take bring their own offset via argument, so they ignore the shared one?
<Regenaxer>
yes, they are an atomic lseek+read
<beneroth>
and of course they are atomic, in opposition to making two separate kernel calls like lseek(); read();
<beneroth>
ok
<beneroth>
related: the file lock/opening flags (e.g. exclusive read and/or write access vs. other process my use the file too) is, beside the files integrity, to protect this shared offset value?
<Regenaxer>
There is no problem if each child does its own (in "file" ...
<beneroth>
yeah, because it is not reusing the shared descriptor
<beneroth>
ok
<Regenaxer>
No, it does not protect the position
<Regenaxer>
offset
<Regenaxer>
locking won't help here
<beneroth>
yeah I mixed it up, the offset is process space, not kernel space. but it is shared in case of inheritance.
<beneroth>
that is the essence?
<Regenaxer>
well, the offset *is* in the kernel
<Regenaxer>
in the file table entry
<beneroth>
yeah, but managed process
<Regenaxer>
but these entries are shared by children
<Regenaxer>
T
<beneroth>
ok
<Regenaxer>
file descriptors
<beneroth>
s/managed process/managed per process
<beneroth>
ok
<Regenaxer>
they point to the kernel file tables
<Regenaxer>
confusing indeed
<beneroth>
and this sharing can not be disabled with an argument to the fork call or something like that?
<Regenaxer>
you can call dup2()
<Regenaxer>
to make it private
<Regenaxer>
a new FD
<Regenaxer>
hmm, no
<beneroth>
ok, so you could, in the pil (fork) implementation, just after the fork call to kernel, call dup2() on all file descriptors the child holds, this would make the pil user code save, no?
<beneroth>
might be needless overhead in some cases, I understand
<Regenaxer>
dup2: and thus share file offset
<Regenaxer>
no, I was wrong
<beneroth>
Same open file (or pipe)
<beneroth>
Same file pointer (both file descriptors share one file pointer)
<beneroth>
Same access mode (read, write, or read/write)
<beneroth>
the offset is in the file pointer or in the fd?
<Regenaxer>
the fd is just a number
<Regenaxer>
the file pointer is stdio
<Regenaxer>
it is the "file entry"
<Regenaxer>
in the kernel's file table
<Regenaxer>
There are some diagrams in Stevens
<beneroth>
yeah right
<beneroth>
wikipedia says it nice in the first sentence: "In Unix-like operating systems, dup and dup2 system calls create a copy of a given file descriptor. This new descriptor actually does not behave like a copy, but like an alias of the old one."
<Regenaxer>
According to Stevens a file table entry has: Flags, offset, and V-node ptr
<beneroth>
well ok
<beneroth>
so after fork kernel call, get those values from each fd, close each fd, re-open each fd with same flags, seek back to the offset
<beneroth>
real copy
<beneroth>
no alias
<beneroth>
no?
<Regenaxer>
You mean to do that in user space?
<Regenaxer>
or if the kernel does it?
<Regenaxer>
In any case this would be a huge overhead
<Regenaxer>
Stevens is really good on that
<beneroth>
in user space, but not pil userspace. within implementation of picolisp (fork), or maybe have two variants of (fork) in picolisp, one like the current and one which might be a bit slower but does that stuff to be less prone to pil user code errors, so that with that variant the old @misc/reverse.l pil code is safe
<Regenaxer>
No good idea
<Regenaxer>
These are border cases
<Regenaxer>
I don't even know all open files explicitly
<Regenaxer>
Would need to parse /proc/self/fd or so
<Regenaxer>
But things might be buffered already
<Regenaxer>
so the context is lost
<Regenaxer>
And between parent and child this is wanted and needed
<beneroth>
yes, but this are border cases which you DO NOT EXPECT AND DO NOT NOTICE as a pil programmer. because it can be safe by accident (short code). and there is no easy guideline to get it right as pil programmer, short of completely understanding the inner workings both of (fork) and posix
<Regenaxer>
only for *other* children
<beneroth>
hm
<Regenaxer>
Can you give any example where this is a danger?
<Regenaxer>
except these forks in interpreted code?
<Regenaxer>
ie where two children read the same FD?
<Regenaxer>
I can't think of any
<beneroth>
I just want our innocence back, so the behaviour is again as the reference says for (fork). if this slipped you and me for years it is not sane to expect people being aware of it. unless there is a big read warning in the description of (fork), but that would also not be nice xD
<beneroth>
hm
<beneroth>
stuff like the @misc/reverse.l ?
<Regenaxer>
It slipped because it is not normal usage :)
<Regenaxer>
yes, like reverse
<beneroth>
well, according to the current picolisp documentation about (fork) this is normal usage :P
<Regenaxer>
And once you know it, you don't do it
<Regenaxer>
no
<Regenaxer>
the ref doesn't do that
<Regenaxer>
(unless (fork) (do 5 (println 'OK) (wait 1000)) (bye)) is all right
<beneroth>
"Forks a child process. Returns NIL in the child, and the child's process ID pid in the parent." nothing else there. so I take from it that I only have to take care of the return value and I'm fine, no other side effects to be expected.
<beneroth>
ah I talk about the description, not the example, T
<Regenaxer>
The description is all right too
<Regenaxer>
It does not talk about reading in other children
<Regenaxer>
or writing
<Regenaxer>
Though writing in append mode is good
<beneroth>
yes. when it is missing, I would expect this to work like it does when no (fork) is involved.
<beneroth>
that is why I wrote code the same as your misc/reverse.l
<Regenaxer>
yeah
<beneroth>
I would like to have something in the reference which makes me aware that I cannot write stuff like misc/reverse.l without further considerations
* beneroth
is not against the current, optimal (fork) implementation
<Regenaxer>
It is a Unix implementation, which we cannot change
<beneroth>
currently, (fork) looks more easy and risk-free than it is :)
<beneroth>
T
<Regenaxer>
Well, in pil it is always different if you operate on the current stream
<beneroth>
but easy-to-make-errors should be easy-to-notice. either because they reliably explode in your face (like segfault when you try to execute a symbol) or just-not-work.
<beneroth>
yes, and (fork) might interfere with the normal picolisp way, depending on how it is used
<beneroth>
T
<Regenaxer>
But explaining such things in the ref is difficult
<beneroth>
at minimum: article and link to that article in reference of (fork)
<beneroth>
T
<Regenaxer>
I would not say "(fork) might interfere with the normal picolisp way"
<Regenaxer>
it does not
<beneroth>
well your (here) might not where you expect it to be, no?
<Regenaxer>
It needs understanding of process environments
<Regenaxer>
yes, (here) or (read) or (line) etc
<beneroth>
yes. but that this understanding is required is not transparent without a clear hint in (doc 'fork).
<Regenaxer>
What would you hint?
<beneroth>
yes, so (here) works fine as always. I add for some reason some (fork) before it, and now it suddenly doesn't reliably work anymore
<Regenaxer>
You need to explain how file descriptors are inherited
<Regenaxer>
an general Unix lesson
<Regenaxer>
You have these effects also with the console
<joebo>
fascinating read!
<Regenaxer>
stdin
<Regenaxer>
stdout
<joebo>
I've worked at the (fork) level before so I'm largely following along
<Regenaxer>
They garble up if you accidentally fork on the REPL
<Regenaxer>
Hi joebo!
<beneroth>
it's difficult. something like (to be added in (fork) reference): Through (fork) the child inherits open file descriptors, when working with inherited file descriptors some possible side-effects have to be taken into considered, read on here: link
<yumaikas>
o/
<beneroth>
shorter preferred, of course
<beneroth>
hey joebo, yumaikas o/
<joebo>
hi Regenaxer , hi beneroth :)
<Regenaxer>
I don't want to explain general Unix stuff in the pil ref
<Regenaxer>
Try : (fork)
<Regenaxer>
Hi yumaikas
<yumaikas>
Hi.
<beneroth>
ok. well what is term for that "inherited file descriptor" stuff then?
* yumaikas
is curious as to what the (fork) is intended to be beyond standard C fork()
<beneroth>
yumaikas, it is about standard C fork. the issue is there are some catches with already-opened file descriptors
<Regenaxer>
yumaikas, yes, but pil's fork does also set up some IPC
<Regenaxer>
not the problem here
<beneroth>
T
<Regenaxer>
The wrong assumption was in the first place that code in a stream behaves like code in an sexp
<Regenaxer>
This is mostly the case
<Regenaxer>
but not always
<tankfeeder>
cusip task done
<Regenaxer>
BTW, I observed similar effects in bash scripts
<joebo>
it sounds like the issue is: the forked child and the parent may share the same file descriptor and the child may advance the position of the fd which could cause problems on future forked children
<joebo>
is that a correct summary?
<Regenaxer>
tankfeeder, looks good!
<Regenaxer>
joebo, perfect
<joebo>
Regenaxer: thanks
<Regenaxer>
on future or past forked children
<joebo>
yep, makes sense
<Regenaxer>
and the parent too
<Regenaxer>
in case the parent reads it too
<joebo>
right
<Regenaxer>
The offset is global inside the family
<beneroth>
so when multiple processes operate on a file descriptor which is shared by multiple processes due to forking.
<Regenaxer>
Harder to explain than it is
<Regenaxer>
yes
<beneroth>
yeah I just managed to explain the issue to my pil coding partner in a few sentences
<Regenaxer>
cool!
<Regenaxer>
clever partners ;)
<beneroth>
he agrees with you: no pil issue, so should not necessarily into pil reference
<Regenaxer>
ok
<Regenaxer>
An article explaining all these things would be best
<beneroth>
T
<Regenaxer>
Perhaps other Unix-dependencies too
<beneroth>
and I still tend to think that the picolisp documentation should not just explain what the pil stuff does but how to use it correctly
<Regenaxer>
true
<beneroth>
difference of "documentation of what is" and "documentation of how to use"
<beneroth>
especially with the focus on practicability the second seems to me more important than the first one.. for complete transparency one can easily read the source, especially as pil source is so accessible
Regenaxer has left #picolisp [#picolisp]
Regenaxer has joined #picolisp
<beneroth>
Regenaxer, T. the difference between Wiki and reference is: reference is atomic, comes with distro. Wiki might be dead/moved. so essential stuff should go into reference, e.g. like form.l documentation also did
<Regenaxer>
grr
<beneroth>
np, you missed nothing
<Regenaxer>
ok :)
<Regenaxer>
^D is too easy
<beneroth>
:)
<Regenaxer>
tankfeeder: cool
<beneroth>
I wondered if you never got tired of doing F-Q all the time? I mean when switching between touch and keyboard activity on tablet. depends heavily on the application and is probably no issue in termux or pilbox..but when using penti as a keyboard for everything...
<Regenaxer>
I'm happy that you continue with rosetta
<beneroth>
T, I second this. good job tankfeeder
<Regenaxer>
I never use the other keyboard
<Regenaxer>
F-Q to close?
<Regenaxer>
ah
<Regenaxer>
I just press outside
<beneroth>
I still love how a CS paper which did some study using rosetta data listed picolisp as one of the languages with the most solutions, even when the paper then focused on mainstream languages for its study.
<Regenaxer>
press one finger outside a circle, then do whatever with oher fingers
<beneroth>
Regenaxer, that doesn't close my keyboard. does this depend on the app or what=
* beneroth
tries out
<Regenaxer>
It does not close, just hides
<Regenaxer>
then touch events go through
<beneroth>
yeah I mean hide
<Regenaxer>
it does not hide?
<beneroth>
ah I see what you mean!
<Regenaxer>
It should hide the circles, and if keeping pressed and moving up/down shoud change the brightness
<beneroth>
press one finger outside of penti circles, AND keep it touched, then touch with other finger(s)
<Regenaxer>
yes
<Regenaxer>
And if keep touched and move up
<beneroth>
ok. something learned. THX
<Regenaxer>
does it change brightness?
<Regenaxer>
if not, you must enable the permission
<Regenaxer>
"system" something
<beneroth>
yeah that I did.
<Regenaxer>
ok
<Regenaxer>
You can also copy/paste this way
<beneroth>
so, I have to take some notes from our conversations today.
<beneroth>
glad this issue got resolved so quickly.
<Regenaxer>
press outside, then select with another finger
<Regenaxer>
:)
<Regenaxer>
or copy from the log?
<beneroth>
still no update on the replica suspicion, right? I haven't had yet the time to look into it or even install checks for it
<Regenaxer>
I'm testing
<Regenaxer>
I did a minor change on the open port
<Regenaxer>
Looks good so far
<Regenaxer>
I killed my router from time to time to see if the hang still appears
<beneroth>
I have a regular backup of the latest snapshot, so I figure I'm ok with the risk of replica being faulty until I can have a closer look on this
<beneroth>
ok
<Regenaxer>
I suspect that if the port is still open, httpGate starts a new connection
<Regenaxer>
Old:
<Regenaxer>
(loop
<Regenaxer>
(let (P (port *Port) S (listen P))
<Regenaxer>
(close P) (http S)
<Regenaxer>
(close S) ) )
<Regenaxer>
New:
<Regenaxer>
(let P (port *Port)
<Regenaxer>
(let S (listen P)
<Regenaxer>
(loop
<Regenaxer>
(http S)
<Regenaxer>
(close S) ) ) )
<Regenaxer>
Could not reproduce the hang with that yet
<Regenaxer>
Will release it after more testing
<Regenaxer>
Let me test, Off for now
Regenaxer has left #picolisp [#picolisp]
Regenaxer has joined #picolisp
<Regenaxer>
Seems OK
<beneroth>
the last I knew was afaik that you couldn't replicate it?
<beneroth>
s/replicate/reproduce
<beneroth>
so you could?
<Regenaxer>
it is difficult
<Regenaxer>
depends probably at which moment the connection is interrupted
<Regenaxer>
does not happen so often
<Regenaxer>
So I force it now once a day, when people are working
<beneroth>
ok. so you found a rough way to reproduce it ^^
<Regenaxer>
I hope so :)
<Regenaxer>
But the change makes sense to me
<Regenaxer>
If httpGate tries to reconnect, because the remote ssl tries to resend after the interrupt, it starts a new replica if the port is not open
<Regenaxer>
So I keep the port open now. Makes more sense anyway
<beneroth>
ok
<beneroth>
in my setup both servers are in (different) data centers. so small interrupts are probably less likely than with your connection.
<Regenaxer>
right
<Regenaxer>
And I have some leak in my router
<Regenaxer>
it hangs after some time
<Regenaxer>
memory leak or so
<Regenaxer>
Switching off and on resets it
<beneroth>
which vendor? FritzBox? D-Link?
<Regenaxer>
I have to do every few weeks
<Regenaxer>
yes, Fritz
<Regenaxer>
an old one
<beneroth>
updated one?
<Regenaxer>
no
<Regenaxer>
very old meanwhile
<beneroth>
they're pretty good, but should be updated. they also had security problems.
<beneroth>
ok
<Regenaxer>
Telekom offers VDSL here soon, so I wait
<beneroth>
yeah I have to get a new router too. currently using a MikroTik, pretty good and quite advanced stuff. actually too many configuration abilities of my understanding.
<beneroth>
before I used a D-Link, which could do NATting and DNS and hairpinning
<beneroth>
but that D-Link broke. and D-Link is horrible insecure apparently.
<Regenaxer>
T
<beneroth>
FritzBox has not the hairpinning configuration options I require afaik.
<Regenaxer>
No idea, I did not really pay attention on the developments
<beneroth>
"Hairpinning is where a machine on the LAN is able to access another machine on the LAN via the external IP address of the LAN/router (with port forwarding set up on the router to direct requests to the appropriate machine on the LAN)."
<beneroth>
ok
<Regenaxer>
I see
<beneroth>
very usable with a Laptop which is used inside and outside of LAN
<Regenaxer>
But you could also go "out" and back?
<Regenaxer>
Too slow?
<Regenaxer>
I do that sometimes
<beneroth>
well either you would have to use two different target IPs (and different target SSL hash for same server)
<Regenaxer>
Not sure, here it works
<beneroth>
or it doesn't work, unless you route with a third server which is always in the outer net.
<beneroth>
afaik
<beneroth>
I forgot.
<beneroth>
I only learn the network stuff so far as I need it when I need it ^^
<Regenaxer>
I go to this server via 10.10.12.7 or 7fach.info
<beneroth>
yeah, hairpinning is when you can use 7fach.info always, even when you are within the LAN
<Regenaxer>
using different scripts
<Regenaxer>
7fach when I'm away
<beneroth>
yeah, it is to abolish the different scripts
<Regenaxer>
but it works locally too
<Regenaxer>
true
<Regenaxer>
I have only two :)
<Regenaxer>
one is "k" and one is "K"
<beneroth>
I have multiple virtual servers in my LAN, so I have the same issue multiple times :)
<Regenaxer>
"K" is the outside one
<beneroth>
k ;-)
<Regenaxer>
Yes, if they are all reachable from outside
<cess11>
Is the demo app in picoLisp.tgz broken? When I try to run it it doesn't send any data after login and it looks like the landing page doesn't fully load.
<cess11>
Maybe httpgate is running. Would that cause symtoms like that?
rob_w has quit [Quit: Leaving]
<beneroth>
should not be broken afaik.
<beneroth>
afaik httpgate must be running so that it works.
<cess11>
It doesn't load properly, at login it seems to accept admin/admin anyway but doesn't return data or 'msg that someone logged in.
<cess11>
Rebooted the machine and tried again so there's no httpgate running. I'm using the Debian system install version of picolisp to run the sources, which usually works fine.
<cess11>
Looked in CHANgES for something about adm.l but didn't see anything. Any other ideas? I think it has been acting up for some time but not sure how long, mostly stick to the REPL when I use it for something or other, but it worked fine a couple of weeks or so ago.
<cess11>
Seems it is that particular machine.
<cess11>
It was firewalled, it seems.
<Regenaxer>
ret
<Regenaxer>
cess11, does it mean it runs now?
<Regenaxer>
8080 was blocked?
<cess11>
No, it was the high number ports that were filtered, as well as 80 and 443.
<Regenaxer>
OK
<cess11>
So yes, it runs, it was entirely a false alarm, or as swedes say, 'skit bakom skaftet'.
<Regenaxer>
:)
<beneroth>
gesundheit
<cess11>
Been using '(in '(w3m Url) (until (eof)(processor (line] expressions recently, quite convenient way to access web data, like open data CSV or similar.
<beneroth>
why not (client) from @lib/http.l ?
<cess11>
It needs a separate handling of SSL.
<beneroth>
T
<cess11>
Which could be done with nginx or httpgate, I think, but I haven't had the time to try it out yet.
<beneroth>
sounds more like a use case for stunnel, openssl s_client or socat
<beneroth>
well I will look into this web scrapping topics soonish.
<beneroth>
My next side-project is a personal browser-proxy. but alas, finding the time for it is an issue.
<beneroth>
Mozilla is going completely nuts.. killing their Plug-Ins (new API) and detach from their earlier ethic values (yeah even more, now they even start to bundle their downloads with spyware)
<beneroth>
the rest doesn't fit my surfing habits
<cess11>
I think it depends on whether it is just one process asking for SSL at a time.
<beneroth>
it is all so horrible.
<cess11>
But I'm unsure, only used stunnel to seep out of networks that wanted me to 'authenticate' with some MITM cert.
<beneroth>
cess11, yeah, using a webserver as proxy for that might be a nice idea!
<beneroth>
well I would like to try a whitelist-only approach for certs.. meaning building a my own whitelist instead of using the ones shipped with browsers
<beneroth>
but maybe that is too insane
<cess11>
SSH does it, no?
<beneroth>
does what?
<beneroth>
maintain their own certs? yeah
<cess11>
One gets to approve every new cert, at least it looks like it when connecting to a new host.
<beneroth>
yes. though those are also often unsigned certs, so it is anyway "trust-on-first-connection"
<cess11>
I don't think it is insane, rather fairly sane.
<cess11>
True.
<beneroth>
question is if it is manageable. if it isn't, more tooling is required so the decisions are less and more highlevel. but this "trust Mozilla, trust Google, trust foo" is not really reliable.
f8l has quit [Ping timeout: 240 seconds]
<cess11>
I think it would be. I'd rather like to accept or deny every cert that every ad is trying to install in my browser, sometimes failing and explicitly allowing me to deny.
<beneroth>
my project idea has already grown into a mix of personal browsing features (e.g. my over-usage of many open tabs and only bookmarking what I already checked out) and secondly I think about storing a lot of data and put in into a pil DB for later mining. e.g. to do web tech usage statistics, or also neat stuff like "has website/news article being altered while looking the same" etc
<beneroth>
would also give some food for testing my pil DB stuff with larger datasets.
<beneroth>
tankfeeder, so short :D
<beneroth>
tankfeeder, where comes 'luhn from ?
<tankfeeder>
from already implemented task
<tankfeeder>
its ok
<cess11>
I just need some gui and more parsing functions, and I'm sort of there but using the usual backends, w3m, curl, wget.
<beneroth>
cess11, I'll keep you updated. so far I haven't even started.
<beneroth>
just many ideas and some concepts how it should work.
<cess11>
I think the tricky part is deciding what and how much to cache.
<beneroth>
I intend to make it a Web App, so can be used by any browser, and if everything fails (parsing-wise) the browsed website can be opened directly
<beneroth>
T
<cess11>
I think I'll make a pil-tiny of that 'in function.
<cess11>
I'll probably make mine a terminal application first, using log files for communicating most of the information gathered, then build the web UI when I know what would be better shown there.
<beneroth>
ok, also a good approach.
<beneroth>
an important goal for my project is a bookmark/website database, so that is a bit different.
<beneroth>
let's see what comes out of it and share insights :)
<cess11>
Yeah, logging visited web pages and keeping a flexible bookmarking system is a good idea.
<cess11>
The script name is a terrible joke.
<beneroth>
nice comments, very explaining :)
<beneroth>
and Regenaxer variable naming style ^^
<beneroth>
cess11, first line in CSV file is headers I assume. If so, I usually parse them / compare them with what I expect. Simple check to abort when the file format changed or wrong file got used for some reason.
<beneroth>
typo in last comment, it should be (mkCSV> ...) not (mkCSV ...) in both instances
<cess11>
For the most part I do imports manually so I look at it and either use it or just 'line it away.
<cess11>
Well spotted, thanks.
<cess11>
But it would be nice to automate more in the long run, I think. Not like CSV:s are going away soon.
<beneroth>
I find CSV the best format short of sexprs
<beneroth>
human-readable, well-compatible, easy to write & read, human-editable with libreoffice/excel, nearly no overhead, fits for all data except nested stuff
<beneroth>
in theory XML could be better but in practice it is (nearly) always bloated, overly complicated, and often not even correctly used (per XML standard, e.g. ordering should not be important) but more like a abhorrent CSV with tag syntax
<cess11>
Yeah. Fast as hell too, and nested structures can be achieved by separating into several files/sets so if one knows what one is doing XML is pretty useless unless one needs to send it to something that only talks XML.
<beneroth>
exactly
<beneroth>
the problem is often is XML deployed by people not knowing what they're doing, just trying to imitate
<beneroth>
at least that is what I saw
<cess11>
Yar.
<cess11>
A few times I've seen decent XML use, mostly in configuration I forgot where it was where key=value didn't fit and one fiddled with it manually.
<cess11>
It is kind of human readable, slightly more so than JSON in my opinion.
<beneroth>
JSON is just XML with less bloaty overhead I find :P
<beneroth>
and in terms of datatypes it is not so sufficient
<beneroth>
bwahahaha
<beneroth>
just (by unrelated news reading) I stumbled upon this:
<cess11>
For machine parsing I find JSON fine, I just don't like to read and manually write it.
<cess11>
At least the consumer types I've seen. I wouldn't call rkhunter or Nagios BS, but they sure are limited.
<cess11>
T
<beneroth>
I only know nagios as monitoring system, not as av
<cess11>
It can keep track of what files have changed and whether they conform with hashes.
<cess11>
At least I think it does, might be something else I'm thinking of.
<beneroth>
the problem is AV which claims to protect you, when all it can be is some tooling to find problems after the damage probably already happened.
<beneroth>
and one attack not prevented is all that is needed.
<beneroth>
see WannaCry. see NotPetya.
<beneroth>
AV is mostly just money and system resources thrown out of the window for nothing.
<beneroth>
get correct backups and educate your team.
<beneroth>
the irony is at least the IT departments in companies are even aware that it is just snakeoil, but they have it for non-technical reasons like "looking professional to people who have no clue" and "insurance"
mtsd has quit [Read error: Connection reset by peer]
mtsd_ has joined #picolisp
mtsd_ has quit [Read error: Connection reset by peer]
<cess11>
Regenaxer: That's fantastic. Thanks.
<Regenaxer>
:)
gko has joined #picolisp
<beneroth>
Regenaxer, bin/ssl does not do any TLS Cert checks, or does it?
<Regenaxer>
It does
<Regenaxer>
chect cert you mean?
<beneroth>
yeah
<beneroth>
is it valid, signed by CA blabla
<Regenaxer>
It does not check if you negate the port iirc
<Regenaxer>
-443
<beneroth>
hu? what has that to do with the port?
<Regenaxer>
it *uses* 443 then
<Regenaxer>
the '-' is just a flag
<beneroth>
yes. what has port usage to do with server cert validity ?
<Regenaxer>
I used it for local tests
<Regenaxer>
nothing
<Regenaxer>
just a flag
<Regenaxer>
(in '("@bin/ssl" "blockchain.info" -443 "de/ticker")
<Regenaxer>
uses 443, but does not check
<Regenaxer>
I used it for local tests with self-genrated certs
<beneroth>
my point is: would @bin/ssl detect a MitM attack with a obviously faked SSL cert ?
<Regenaxer>
I hope so
<Regenaxer>
openssl should do
<beneroth>
ok
<Regenaxer>
I expect that at least
<beneroth>
I'm not sure how much openssl does in itself without explicitly telling it so
<Regenaxer>
if (cert = SSL_get_peer_certificate(ssl)) {
<Regenaxer>
X509_free(cert);
<Regenaxer>
if (SSL_get_verify_result(ssl) == X509_V_OK)
<Regenaxer>
return sd;
<Regenaxer>
}
<Regenaxer>
58,23/296 19% in src/ssl.c
<beneroth>
yeah thanks
<beneroth>
I research now what it does
<beneroth>
sorry for doubting, just curious
<Regenaxer>
Thanks. I don't remember well ;)
<Regenaxer>
no, this is very good
<beneroth>
I'm not doubting the usefulness of @bin/ssl :)
<Regenaxer>
yeah
<Regenaxer>
But multi-eye checks are needed
<beneroth>
T
<beneroth>
nothing wrong with not doing every possible security check, but one has to be aware then for what one uses the stuff
<Regenaxer>
T
<beneroth>
I'm very interested in using @bin/ssl. though I might like feed it a customized CA store and get more insight than usually needed
<beneroth>
understanding @bin/ssl would be a good learning step :)
<Regenaxer>
bin/ssl is rather specialized, mainly for mirroring. Now I use 'socat' to check my mails
<beneroth>
Why did you stop using bin/ssl for the mail connection? (IMAP I assume?)
<Regenaxer>
No, pop3. I don't use it cause bin/ssl does an implicit GET
<Regenaxer>
So only useful for https
<beneroth>
ah right
<Regenaxer>
could be changed
<Regenaxer>
but socat is perfect for such tasks
<Regenaxer>
and flexible
<beneroth>
well better use something tested than do it yourself, especially in security
<Regenaxer>
yep
<Regenaxer>
but the security-relevant parts are tested (openssl)
<Regenaxer>
(I hope)
<beneroth>
yes. though it fails sometimes.. also because openssl appears to be legendary messy
<Regenaxer>
The real, original purpose of bin/ssl is the asynchronuous replication
<Regenaxer>
true, much too complicated API
<beneroth>
I'm not sure yet if that very function call is doing all the stuff you would expect it to do. I suspect it only checks if the cert is a valid cert (as in file format), not if it has a valid signature etc
<beneroth>
but I don't know
<Regenaxer>
hmm, would be pretty useless then
<Regenaxer>
It must be able to do it, the question is whether I use it correctly
f8l has joined #picolisp
<beneroth>
"For compatibility with previous versions of SSLeay and
<beneroth>
OpenSSL a certificate with no trust settings is considered to be valid for all purposes."
<beneroth>
my current feeling is: it surely can check the certificate chain it receives from the server
<Regenaxer>
good
<beneroth>
it can check the CA root cert with a local store of trusted CAs. the cli interface has arguments for this.
<beneroth>
ah this might be
<beneroth>
SSL_CTX_set_default_verify_paths
<beneroth>
I'm going very naive and stupid into this. sorry :D