i sure hope nobody figures out that kitsune tails save files are just a proprietary binary version of json thats fairly human readable and so easy to reverse engineer and hacks them for their own purposes 🥺
@eniko For the only (very bad) game I made as an adult (so not counting the crap I made as a kid on a C64), I just used straight-up s-expressions as my save format. :P
@eniko@ouro@carbonacat you may have questions about why I made certain decisions in the standard for this encoding. For all inquiries, I refer you to the first three words of the name.
@eniko
I disagree (it's not that worse than JSON, it just got overblown by getting squeezed everywhere, even when it made no sense, with the result that now everybody avoids it like the plague even where it makes sense), and I'm ready to guess you're effectively rolling your own EBML ;-)
"but eni why didn't you just use one of the existing binary json encodings"
because i looked at them all several years ago and they're all shit, actually. they're incredibly overcomplicated and in almost all cases are not actually even 1:1 compatible with regular json which like, wow, you had one fucking job
the closest i found was Universal Binary JSON which still makes several bizarre choices, not the least of which is that it uses big-endian because apparently its still 1980 and we're all using motorola 68000 cpus, but it's also so poorly maintained that several times its website got somehow hijacked by malicious actors so that makes it really hard to use or recommend
@joshix technically big endian is "network order" because, well, nobody knows its just what they chose for TCP/IP. but given that there are basically no big endian CPUs used in consumer computing (or most computing of any variety) and even bi-endian CPUs are run almost exclusively in little endian mode it makes no sense to make a new format that uses big endian in the modern day like the UBJson folks did. it just adds a bunch of byte shuffling overhead and makes it harder to read and write (for example C# doesn't have a built in way to read/write big endian values on a little endian system)
@joshix i have to disagree. i dont know about you but i read from left to right, not right to left, and i think from small numbers to large numbers. this is not compatible with big endian and in fact the first byte of a big endian number gives me no meaningful data if i don't know what type of integer is, while i can still get the small portion of any integer by reading the first byte in little endian
@eniko A random JSON-like format is still much better than the Bad Old Days, where people would make on-the-fly file formats which were basically just dumps, where position in the file determined what each byte meant.
Also, vary the offsets of all meaningful bytes based on the exact game state saved.
Store some values big-endian, others little-endian.
Maybe add a non-standard compression to make it more difficult to read.
None of that was done to obfuscate. It just kinda happened that way.
.. my save games are also a binary JSON format, but I narrowly managed to accept the things I don't like about messagepack enough to use that instead of making my own
(on the other hand, I did not manage to accept the things I don't like about the C++ messagepack libraries so I made my own)
@mort tbh it's super simple. objects are encoded using { and } char bytes, arrays with [ and ], and then values all have a prefix like K for keys, S for strings, T for true, F for false, B for unsigned byte, i for signed integer, or 0-9 for those values, followed by the actual value if necessary
@mort i like it! although i personally have some concerns about "Floats and doubles are distinct" and "Objects are ordered" since that fails my round-trip compatibility with json test
@mort this also gives me an idea for taking this further and making it truly non-binary by adding a LEB128 string size for strings so you can skip the null terminator and encoding all "binary" values as hex instead 🤔
@eniko Yeah, I get that. I guess I could define float -> double conversions and int -> double conversions to be semantics-preserving...
However I like ordered objects :( My parser is streaming and I sometimes need one key of an object before another key, so I just declared objects to be ordered
@mort yeah i ran into that same issue in the past. what i tended to do to keep things json compatible was an array with an object inside. that way the array becomes an ordered "header" for the object, and its still round trip compatible so instead of like
{ "a": 1, "b": true, "someProp": "haha" }
i'd do it like
[ 1, true, { "someProp": "haha" }]
of course you do lose the keys on the ordered items but for things that are ordered keys shouldn't be necessary anyway
@eniko That's definitely a solution, though I feel like it locks the format down a bit more; you can't just add more things to the header without breaking stuff. You also kinda lose the self-documenting property of having keys.
Another approach I've considered is to parse everything independently up-front, into some intermediate variables if necessary, and then fix up things afterward.
I guess I could add some weaselly language to the spec: implementations may treat order as significant 🤔
@eniko I like the sound of that a lot I think. Interesting that you encode keys and strings differently, I guess that helps when inspecting the file manually; you can reset your mental parser at every "K<string>".
How do you encode keys/strings? Are they 0 terminated?
@eniko I threw together a msgpack -> enikoJSON, but with 'p' for positive LEB128-encoded integer, 'm' for negative LEB128-encoded integer, 0 terminated keys/strings, 'f' for 32-bit float, 'd' for 64-bit float, plus a 'b' for length-prefixed arbitrary binary data and 'N' for null: https://p.mort.coffee/M9S.cc
Files are slightly smaller with msgpack than this format (5927B vs 6264B; 4951B vs 5043B gzipped). But it's sooo much simpler. Hm.
Add comment