<razzy>
what is hard is to test quality of encryption
<tankf33der>
start point
<Regenaxer>
the math
<Regenaxer>
and to be sure not to have security holes
<Regenaxer>
all no issue with compression
<razzy>
tankf33der: well 10 years of standard human reading
<razzy>
thx :]
<razzy>
on second thought. not very well documented
<razzy>
i know what i want to do, i just need to write it :]
<Regenaxer>
perfect ;)
<tankf33der>
razzy do you a pseudocode?
<tankf33der>
do you have a pseudocode?
<tankf33der>
razzy: looks like you fan of silicon valley tv series
<tankf33der>
razzy: you know there is also compression algo in the script, right?
<razzy>
i feel that there should by way less core functions in picolisp. there should be lik 50 of them and the rest in separate library :]
<razzy>
it would ease me a start :]
<Regenaxer>
Would be possible, but I see no advantage
<razzy>
tankf33der: i do have pseudocode in my papers:]
<Regenaxer>
ignore funs you don't neet atm
<Regenaxer>
need
<razzy>
Regenaxer: the non-essentials are making noise to cut through
<Regenaxer>
But then the libs would too
<tankf33der>
writing compression as first code on new language is extreme like base jumping
<Regenaxer>
Everything in the base system is essential
<Regenaxer>
And the base is easy to browse with (doc 'doc)
<razzy>
tankf33der: i jumped long time ago :]
<tankf33der>
razzy: ok then
<razzy>
now it is just fallout
<razzy>
tankf33der: problem is, i keep changing my pseudocode to fit what i can do in picolisp
<tankf33der>
this is not a proble
<tankf33der>
show pseudocode
<razzy>
tankf33der: working on it
<tankf33der>
ok
freemint has quit [Ping timeout: 240 seconds]
<razzy>
tankf33der: well for start i picked up this.https://ptpb.pw/36CN pay atention to first commented part.
freemint has joined #picolisp
<freemint>
back
Lumpi7 has joined #picolisp
Lumpi7 has quit [Remote host closed the connection]
<freemint>
i am working on my first database backed program. Just testing the water so far. It will run from console.
<freemint>
Is there another way to make entities other than 'obj?
<Regenaxer>
yes, 'new' mainly, and 'request'
<freemint>
I am unsure what way to go here, i ask when i am done reading the relevant docs
<Regenaxer>
'obj' is only a convenience function for constant initialization
<Regenaxer>
Internally it all boils down to 'new' anyway
<freemint>
Should i modify the 'T method?
<Regenaxer>
Normally not modify, but override
<Regenaxer>
you mean of +Entity, right?
<freemint>
let me specify
<freemint>
I want to create an own user login system.
<freemint>
Each user has name, hash and pepper
<razzy>
tankf33der: better explained how to count strings in stream file https://ptpb.pw/Rlzh
<freemint>
the pepper is randomly generated each time a new password is set and the hash is set to password concatenated with pepper. The create user function takes name and password but stored, name, hash and pepper
<freemint>
how would i go about that?
<razzy>
tankf33der: basicly frequency analysis of text file
<Regenaxer>
Why is the existing one not enough?
<Regenaxer>
With "pepper" you mean "salt"?
pierpal has joined #picolisp
<freemint>
The existing one is good, but i want my one as a little project to get going with database
<freemint>
pepper is crypto term for per user salt
<Regenaxer>
ok, good
<razzy>
tankf33der: will you help me?
<Regenaxer>
freemint, I see, never heard
<razzy>
i like the terminus technicus
<freemint>
the idea of per user salt (pepper) is to have the attacker not reuse the hash result between users
<tankf33der>
razzy: i will help you only with picolisp
<tankf33der>
gmt+2 working hours
<tankf33der>
bed
<Regenaxer>
tankf33der, so Riga is in the same zone as Berlin? I thought one more east
<razzy>
tankf33der: do you understand second pseudocode? or should i work it differently?
<tankf33der>
berlen -1 hour
<freemint>
Regenaxer how would i create such user? If i were to do it with normal objects i would write an own T method.
<Regenaxer>
You create a random pepper? Then T is fine
<Regenaxer>
Look eg wiki/er.l
<Regenaxer>
in +User T
<freemint>
But database confuses me for 2 reasons. Whether i should use the constructor is unclear and how to fill the database is a little unclear too. new does not take attributes
<Regenaxer>
(dm T @
<Regenaxer>
(pass super 'ca (date) 'new T) )
<freemint>
let me check
<freemint>
what is the @ for in this example?
<Regenaxer>
I do not call the T method a constructor, it is rather an initializer
<Regenaxer>
@ is for more arguments
<Regenaxer>
(new! '(+User) 'nm "Hugo" ...
<Regenaxer>
Each T in subclasses usually adds more properties
<freemint>
so this creates an Timestamp and all arguments are thrown away?
<Regenaxer>
Why thrown away?
<Regenaxer>
it *adds* a timestamp and a "new" flag
<freemint>
oh i did not knew what pass does
<Regenaxer>
yes, passing the args
<freemint>
that makes sense
<Regenaxer>
tankf33der, so you are GMT+3
<freemint>
Will be going to FrosCon this year
<freemint>
i might give a lightning talk about pil
<Regenaxer>
cool!
<Regenaxer>
In St.Augustin again?
<freemint>
yes
<freemint>
What happens if set an already existing value in a key?
<freemint>
is checking for username uniqueness a think that should happen in the constructor ?
<Regenaxer>
Then you have changed the key
<Regenaxer>
I would not check it manually, but rely on +Key
<Regenaxer>
mis> will check
<Regenaxer>
automatically in the gui
<freemint>
i go without gui for now
<Regenaxer>
Then call mis>
<freemint>
what does mis> do normally
<Regenaxer>
It checks for any mismatch
<Regenaxer>
eg data type
<freemint>
on what would i call miss?
<Regenaxer>
Before you set a value
<Regenaxer>
I never called it myself iirc
<Regenaxer>
only gui
<freemint>
ok
<Regenaxer>
But I think better use 'request' or 'obj'
<Regenaxer>
you don't get double keys then
<Regenaxer>
instead it uses the existing object
<freemint>
how do they react to a double key?
<freemint>
ahh
<Regenaxer>
'request' is typical for imports
<freemint>
so i write the T method ignorant of this constraint?
<Regenaxer>
yes
<freemint>
what do you mean by imports?
<Regenaxer>
Importing data, normally CSV
<Regenaxer>
excel or other DBs
<freemint>
ok, why wouldn't you use it for normal data creation?
<freemint>
(like on a web server?)
<Regenaxer>
If you *know* it is new you don't need 'request'
<Regenaxer>
eg a "New" button
<Regenaxer>
then normally an empy object is created
<Regenaxer>
or with some defaults
<Regenaxer>
Then when the user types a name 'mis>' complain
<Regenaxer>
s
<freemint>
ok
<freemint>
my code looks like this
<Regenaxer>
So in the gui objects are empty initially, but an import will hit existing ones
<freemint>
(dm T (nm pw . @ ) (let R (rand) (pass super 'nm nm 'ppr R 'pw (hash (cons pw R)))))
<freemint>
other than it being very hacky . Is that fine?
<Regenaxer>
Better Nm and Pw
<Regenaxer>
otherwise fine I think
<freemint>
so i would create a new user with (new! '(+User) "Mark" "BattreyHorseStapple" )
<freemint>
?
<Regenaxer>
right
<freemint>
how can i check whether there already is a mark who is an user?
<Regenaxer>
'mis>', or just 'db' or 'fetch'
<Regenaxer>
(when (db 'nm '+User "Mark") ...
<freemint>
thanks
<Regenaxer>
(when (fetch '(nm . +User) "Mark")
<Regenaxer>
the latter for +Key
<freemint>
these things are obvious once you did them a few times
<Regenaxer>
'db' works also for non-unique
<Regenaxer>
indeed
<freemint>
also you definitly want to use a if
<Regenaxer>
The question is what to do if "Mark" exists
<Regenaxer>
use the existing one?
<Regenaxer>
then (request ... is better
<freemint>
*confused*
<Regenaxer>
(request '(+User) 'nm "Mark")
<freemint>
that did not confuse me
<Regenaxer>
Then you are always unique
<Regenaxer>
The GUI complains if "Mark" exists, so without GUI you must be sure *beforehand* that the data are good
<Regenaxer>
eg when importing from a CSV
<Regenaxer>
you can't give an error then
<Regenaxer>
So I always use 'request' in imports
<freemint>
i will understand this another day
<Regenaxer>
yes, no big thing
<Regenaxer>
Just experiment
orivej has joined #picolisp
<Regenaxer>
Getting sleepy
<Regenaxer>
I'm off for today
<freemint>
good night
<Regenaxer>
Good night!
<Regenaxer>
afp
<freemint>
good night
<freemint>
oh it works so wonderful
erkin has quit [Read error: Connection reset by peer]
Sveta17 has joined #picolisp
Sveta17 has quit [Remote host closed the connection]
<freemint>
razzy do you have an idea how you want to compress?
<razzy>
freemint: yes very much
<razzy>
freemint: want to help me?
<freemint>
can you explain your basic algorithm?
<razzy>
something between hofman and lzw with some deep learning heuristic
pierpa has joined #picolisp
<razzy>
but for start, i need stemlined frequency analysis
<razzy>
streamlined
<freemint>
how will you the deep learning?
<freemint>
*will you do the
<razzy>
deep learning does not have to be complicated
<freemint>
then you are probably using the wrong word. Deep learning is always complicated
<razzy>
symbols and variables in lists
<freemint>
what do you mean by deep learning?
<freemint>
what do you have in mind for that?
<razzy>
structure several layers of variables that propagates error
<razzy>
and learn
<freemint>
how will you do the derivatives?
<razzy>
there is standard neuron
<freemint>
What activation function do you intend to use?
<razzy>
not very important
<freemint>
Is this function implemented in PicoLisp and if not can you implement it7
<razzy>
if i understand picolisp well enough, yes i can
<freemint>
How will you ensure that the do not need the neural network to decompress?
<razzy>
that is the plan
<razzy>
it is just heuristic to help find good compress
<freemint>
what compression primitives do you have to work with?
<razzy>
what do you mean compression primitives?
<freemint>
the networks gives you an heuristics on what algorithm to choose when compressing?
<razzy>
could be but not yet
<freemint>
what is the network for then?
<razzy>
in broad sense, network suggest which algoritms to use
<razzy>
yes
<razzy>
very broad sense
<freemint>
based on what metrics does the network decide?
<razzy>
freemint: will you help? or you just mining info
<freemint>
does it see the complete stream and if so how does it memorize stuff?
<freemint>
I am trying to help since you seem very confused about what you want to do
<razzy>
i might seem that way :]
<freemint>
(Nobody in there right mind would pick picolisp for this task i think) (no insult) also i am curious why you did choose picolisp for this
<razzy>
i would say that no normal sane person would choose picolisp
<freemint>
yeah
<freemint>
so why did you choose it?
<razzy>
little baggage, lispy enough.
<freemint>
ok
<razzy>
scheme was another choice
<razzy>
or racket
<freemint>
one problem i see is that file the file access functions in PicoLisp are not perfect for that. (on the the other hand scheme would be just as bad and picolisp easier to extend)
<freemint>
If i had to choose i would go with clojure
<freemint>
all the Java libraries for compression would be helpful
<razzy>
freemint: picolisp has wonderfull file access imho
<freemint>
it lacks jumping to a position in a file
<freemint>
(i think)
<freemint>
anyhow what questions do you have?
<razzy>
there is peek, but i did not need that
<razzy>
i do not know how it works
<razzy>
clojure is fucking corporate animall
<freemint>
I would never think in terms of tables in picolisp for counting things
<freemint>
build an idx tree
<freemint>
(doc 'idx)
<razzy>
i want to check my code and rewrite it to be picolisp compliant
<freemint>
tables in picolisp are lists and when you a 60000 picolisp will follow so many pointers to increment an number
<razzy>
i writen table, meant lists structure
<freemint>
yeah, go with a tree when you can.
<razzy>
120k cell lists seems ok
<freemint>
lwzCompress takes an compression-table?
<razzy>
could, depend on implementation
<razzy>
imho
<freemint>
doing one operation on this list of 120000 will take roughly 2ms on my machine
<freemint>
anyway ... optimization comes later
<razzy>
yop
<freemint>
if lwzCompress Takes an compression-table. where will this table be generated?
<razzy>
lzwCompress is bad name. compress generate Comp_table
<freemint>
a comp-table and a compressed file?
<razzy>
Comp-table is compression table with statistics and how to decompress
<razzy>
out file suppose to be compressed
<razzy>
i am not there yet
<freemint>
all you still failing to fill your buffer?
<freemint>
(while 1000>(lenght Buffer)) this line is wrong on so many levels
<razzy>
yes, it is my pseudocode
<freemint>
(doc 'make) is your friend.
<freemint>
What do you intend to do with the buffer?
<razzy>
use it as in section 2.1 and 2.2
<freemint>
do you want the buffer to always be 1000 elements long?
<razzy>
yes i reconsidered 100 long
<razzy>
but normal list is ok
<razzy>
no advantage imho in making it special
<freemint>
does the buffer contain the file or something else?
<razzy>
bits of file
<razzy>
readed over and over and crossover adressed
<freemint>
ok. then i do not grasp what you try to do with the file...
<razzy>
file is GBs long
<freemint>
2.0 is very vague
<freemint>
i know
<razzy>
it is explained in 2.1 and 2.2
<freemint>
no "frequency of repeating characters" is badly worded
<razzy>
for start i want to count frequency of repeating strings
<razzy>
count number of repeating strings
<freemint>
how do you know that a string is repeating?
<freemint>
(without having the complete file in memory)
<freemint>
where does a string start and end?
<razzy>
freemint: 2.1 and 2.2
<razzy>
it reads the file and make some guesses.
<freemint>
so you look for exactly 100 byte long strings?
<freemint>
and look how often they appear?
<razzy>
up to 100 char long strings
<razzy>
yes
<freemint>
so if you read "abha" you try to insert "abha" "bha" "ha" and "a" ?
<razzy>
i try to make table from all of it
<freemint>
what is "all" ? your algorithm is very underspecified
<freemint>
is my abha example right?
<freemint>
(assuming abha is your buffer)
<razzy>
key is, that when string is meeted again, it is solidified. when is not meeted again, it may be replaced with other one
<freemint>
that is obvious
<freemint>
but what (sub-) strings do you try to insert
<razzy>
for start all of them
<razzy>
later deep learning helps
<freemint>
are you aware that if you process your file in chuncks you loose sub-strings?
<razzy>
not if you do not pop strings that you meet for first time
<freemint>
example "abhabhabhabha" processed in chunks of four is "abha" "bhaa" "haab" "aabh"
<freemint>
in an optimal compression scheme we should say the we see abh 4 times
<razzy>
will you write it for me if i write whole example?
<freemint>
write what?
<razzy>
picolisp
<freemint>
well ... i do not want to steal the learning experience but if that helps you ... i would invest 1 hour trying to write atleast
<freemint>
or did you take the algorithm from somewhere else ... maybe that resource describes him better
<freemint>
?
<freemint>
then you would not have to write anything
<razzy>
no, as far as i know, unique
<razzy>
i am writing example
<freemint>
honestly from what i know about compression your approach seems hopelessly naive. 1. Deep learning will not be able to help you since every file is different 2. you are using a fixed length for maximum symbol length 3. the number of substrings explodes with the length of the file 4. you attempt to fix that problem with a maximum symbol length which is choose arbitrarily without proper statistical or theoretical grounding.
<freemint>
But the problem sounds fun, so i see what i can do
<freemint>
razzy why is the example taking so long?
<razzy>
i am tired and read to hofman
<freemint>
do you mean Huffman-Encoding?
<freemint>
"Huffman"
<razzy>
yes, do not know how close i am to it
<freemint>
well i can tell you
<razzy>
you are right about exploding, some heuristic should help there
<freemint>
there are no general purpose heuristics which make it shorter without make compression more bad in many cases
<freemint>
what makes approach different from Huffman is that he has constant word length
<freemint>
your extension to longer symbols brings new problems:
<freemint>
your problem is that your substrings will overlapp
<razzy>
yes
<freemint>
"ablabablab" lab and ab or ablab or abl and ab
<freemint>
will all be good encodings and there is no way how you can recovers that information from a frequency list
<razzy>
solution is to use heuristic, "prefer 2 longest"
<freemint>
first off heuristics are not solutions
<freemint>
heuristics are approximations
<razzy>
well yes
<razzy>
i do not need perfect solution, just better than everyone else
<freemint>
2. any heuristic a neural network can come up with is likely bullshit
<freemint>
3. The level on which you can specify these heuristics is inappropriate for the task
<freemint>
4. you will not Winn the hutter price with that approach
<razzy>
in random example yes,.. but human text or usefull data has rules to them. neural networks pick them up
<razzy>
4. maybe :D
<freemint>
are you ducking kidding?
<razzy>
ducking?
<freemint>
fucking
<razzy>
not really, hutter is good :]
<freemint>
if i were to attempting to solve this prize i would build exploit structural information like XML tags
<freemint>
sorry but you are totally going down the wrong path
<razzy>
we will see
<razzy>
or not
<razzy>
:D
<freemint>
sorry but if you are trying to claim an improvement on that prize, i think i can not help you