<Regenaxer>
beneroth, there is a cool new function in pil64 that might be useful for you
Regenaxer has left #picolisp [#picolisp]
Regenaxer has joined #picolisp
<Regenaxer>
oops
<Regenaxer>
It is called 'blk'
karswell_ has quit [Remote host closed the connection]
<Regenaxer>
allows direct DB file access, on files not necessarily in the current DB
<Regenaxer>
I needed it for my distributed DB application (BTG)
karswell_ has joined #picolisp
<Regenaxer>
I first did all with remote calls, but on the mobile device it it tedious to start so many DB applications
<Regenaxer>
And for read-accesses it is much more efficient to directly access local files than via RPC over TCP
<Regenaxer>
On the main server I start many applications anyway, no problet there with httpGate
<Regenaxer>
But on mobiles I have no httpGate, and would consider it as overkill
orivej has joined #picolisp
<beneroth>
wohoo
<beneroth>
Regenaxer, exciting. thanks for notice!
<Regenaxer>
:)
<beneroth>
yeah that could be handy for advanced locking mechanisms
<beneroth>
and for quick random access :)
<beneroth>
ah
<beneroth>
yes, I can definitively use this.will use it first for a meta-application to manage other pil DBs
<beneroth>
so same use case as yours, reading a not-pool'ed DB
<beneroth>
qsym is also new, eh? nice too
<beneroth>
Regenaxer, is the read atomic? meaning if no read-lock is placed on file, is the proper reading of a single database object (which could be multiple blocks) guaranteed?
<beneroth>
(obviously no guarantees when reading multiple objects in multiple blk calls without locking)
<Regenaxer>
ret
<Regenaxer>
No, 'qsym' is very old
<Regenaxer>
used in remote queries usually
<Regenaxer>
In BTG I use both now, RPC and direct file read
<beneroth>
oh ok.
<beneroth>
yeah there use cases for each
<Regenaxer>
only the remote queries I replaced with direct 'blk' now
<Regenaxer>
T
<Regenaxer>
On the main server, writes are triggered on remote DBs
<Regenaxer>
eg with +Joints across applications
<Regenaxer>
on PilBox, only a single DB is writable, all others are read-only
<Regenaxer>
so 'blk' is perfect
<Regenaxer>
Main reason was that it does not require other processes to run to serve the requests (queries)
<Regenaxer>
So the DBs are 100% bit-compatible on the server and on PilBox, but the handling is different
<beneroth>
I see
<beneroth>
a single (blk) read is atomic? meaning if no read-lock is placed on file, is the proper reading of a single database object (which could be multiple blocks) guaranteed?
<Regenaxer>
yes
<Regenaxer>
It locks the db, and then reads all blocks of that object
<Regenaxer>
Just like normal DB read, only outside of a normal DB
<beneroth>
what if fd2 is not given?
<Regenaxer>
Then it does not lock
<Regenaxer>
eg in a single-process env
<Regenaxer>
Normally in PilBox
<Regenaxer>
I introduced fd2 just now, the version from yesterday needed an explicit (ctl "+db/messe/@" ...)
<beneroth>
ok, so if a concurrent write is happening on that exact object during (blk) read without fd2, (blk) might fail or return garbage?
<Regenaxer>
no
<Regenaxer>
it blocks until the DB is released
<Regenaxer>
the lock
<Regenaxer>
ah
<Regenaxer>
without fd2, yes
<beneroth>
I mean without fd2
<Regenaxer>
yes, sorry
<beneroth>
ok.
<beneroth>
maybe better update the example to also make use of fd2, as this should be the default usage I think ;)
<Regenaxer>
Normally fd2 is from (open "db/app/@")
<Regenaxer>
yes, right
<Regenaxer>
done
<beneroth>
and maybe include 'fd in the list in the ref.html "Other (derived) data types"
<Regenaxer>
yeah, thought so too, using 'cnt' is not very clear
<beneroth>
yeah introducing 'fd is a good idea, even if it is a 'cnt too :)
<Regenaxer>
T
<Regenaxer>
OK, done too
<Regenaxer>
fd2 is some times fd, and sometimes another file
<beneroth>
nice mechanic. I like that fd2 is optional. allows omitting it when not needed (single process) or to substitute the normal mechanics with a custom one (e.g. using another file as a lock-signal, e.g. having multiple lock-files for different parts of the same db-file)
<Regenaxer>
Needs some logic in the application to maintain all these open files
<beneroth>
yeah got it :)
<beneroth>
naturally
<Regenaxer>
Right, a completely different file also may make sense
<Regenaxer>
just like wit (ctl ...)
<Regenaxer>
But ctl has more overhead, as it opens and closes that file
<Regenaxer>
And 'blk' should be as fast as possible
<beneroth>
T
<Regenaxer>
I had it working with 'ctl' yesterday, now I rewrite it for fd2
<Regenaxer>
need another assoc list to hold the open files
<beneroth>
you might want to use (ctl) to manually secure multiple (blk) calls, like (ctl File (blk) (blk) (blk))
<Regenaxer>
right, to get objects depending on each other
<beneroth>
e.g. reading the DB root node which has a property for a general settings symbol (singleton, so no need for a +Entity class), than reading that settings symbol. two (blk) calls.
<beneroth>
T
<Regenaxer>
So ctl still works the same, just omit fd2
<beneroth>
aye
<beneroth>
nice specialist function :)
<Regenaxer>
Then it is not time critical anyway, reading multiple
<Regenaxer>
yeah
<beneroth>
T
<Regenaxer>
I call it in the *Ext mechanism
<beneroth>
I have to finish 1 client application (based on pil DB and my mashina framework), then I will I do some more mashina development :)
<beneroth>
I see. clever
<beneroth>
ha. nice abstraction
<Regenaxer>
yep, (show '{A@20}) shows object '{20} from the other DB
<beneroth>
beautiful
karswell_ has quit [Read error: Connection reset by peer]
<Regenaxer>
And the other good thing is that the remote *DB root object can be used as a Hook
<beneroth>
aye
<Regenaxer>
Most of my time currently went into PilBox anyway these days
<Regenaxer>
Listening radio, and I needed a new alarm clock
<Regenaxer>
My old one is still analog ;)
<beneroth>
ah, it is a pilBox app?
<Regenaxer>
yes
<beneroth>
so you go straight from electronic age to future pil age ;)
<Regenaxer>
T
<Regenaxer>
From FM radio to WiFi
<Regenaxer>
One problem not solved yet is how to do 'dbgc' on these multiple DBs
<Regenaxer>
Until now I had only multiple stand-alone DBs which could be collected independently
<Regenaxer>
But now I have joints and links between them
<Regenaxer>
Same with dbCheck
<Regenaxer>
Seems I opened a pandora box :)
<beneroth>
not necessarily
<Regenaxer>
It is the 'mark' function
<beneroth>
I guess the problem is you can't be sure that the state has not changed when you collected enough data to do some cleaning
<Regenaxer>
It refuses to handle objects outside the DB range
<beneroth>
so I guess you should do it differently, e.g. with timestamping
<beneroth>
ah
<beneroth>
tricky
<Regenaxer>
T
<Regenaxer>
I think if I extend 'mark' it might work
<Regenaxer>
both dbgc and dbCheck
<Regenaxer>
I try tomorrow
<beneroth>
I'm interested in your findings :)
<Regenaxer>
yes, me too :)
<Regenaxer>
It is a very powerful and efficient mechanism
alexshendi has quit [Ping timeout: 256 seconds]
<Regenaxer>
The solution is probably that eg. 'dbgc' must traverse the linked/joined objects in other DBs, but not 'mark' them. This should do, as the other DB's objects are not collected in the sweep phase anyway, they are just ignored
<Regenaxer>
They will be collected when *those* DBs are 'dbgc'd later
<beneroth>
ah
<Regenaxer>
Hmm, no, not sure
<beneroth>
so every DB cleans itself
<Regenaxer>
yes
<Regenaxer>
But I have objects which have no index in their db
<Regenaxer>
so they depend fully on the other instance
<Regenaxer>
I cannot collect these dbs independently
<Regenaxer>
only all as a whole
<Regenaxer>
Let's see, I look at it during the next days
<Regenaxer>
Not in the evening, I cannot do such things late
<beneroth>
you will probably need some kind of two-phase mechanism. first collect, check if collection still is valid (no edits which made them reachable again), sweep.
<beneroth>
or maybe a different mechanism than the one for the single DB.
<Regenaxer>
Before I though of storing references to other DBs in a file. The when that DB is collected, it reads the file and marks those objects
<Regenaxer>
It is a kind of supervising dbgc
joebo has quit [Quit: WeeChat 1.1.1]
rob_w has joined #picolisp
rob_w has quit [Read error: Connection reset by peer]
mario-goulart has quit [Read error: Connection reset by peer]