ascherbaum, 14 days ago

@orsvarn What do you mean with "entry"? Table? Record?

orsvarn, 14 days ago

@ascherbaum probably “record”. Like row in a table.

ascherbaum, 14 days ago

@orsvarn Every row is a file on disk? How is that faster?

orsvarn, 12 days ago

@ascherbaum because I wouldn’t have to load everything to do anything.

• To read 20 records I could load just those 20 files, that’s fast!
• If I want to add a record I just add a new file, that’s fast!
• If I make an edit I just edit a single tiny file, that’s fast!

In my use case I would never need to load more than like 150 records regardless of the size of the DB. I think loading many records would be the largest performance drawback of having one file per record.

My current implementation using one JSON file for the entire DB starts getting slow a few thousand records.

ascherbaum, 11 days ago

@orsvarn This leaves more questions open than it answers:

• How do you place records (files) in "tables"?
• How do you know which files to read for a record?
• How do you handle thousands of records or files? This is a lot of syscalls.
• Did you measure performance, compared with something like SQLite?

I somehow doubt that opening 20 files one by one is faster than opening one file and reading the content.

ascherbaum, 11 days ago

@orsvarn As to why JSON is slow, that's a whole different story. If you need to parse the entire JSON every single time, I can see that this doesn't perform well.

orsvarn, 11 days ago

@ascherbaum

• A directory is a table, a file in it is a record.
• The name of the file is the primary key.
• I don't understand what you mean by how I handle it.
• I did not measure performance, this is still hypothetical as I haven't even built this "database" yet!

Someone suggested I check out SQLite, so I'm experimenting with that now, seems perfect for my use case! :)

ascherbaum, 11 days ago

@orsvarn Re "handle syscalls": if you have on file per record, reading the file involves open(), followed by read(). Or mmap() the file.

Since your records are small, you are sending many syscalls for small files to the kernel, which is slow.