@solidangle As someone with an application background I was always astonished as to how little games make use of databases, and instead rely on gigantic single file json or similar. Maybe just a feeling but I think databases (esp. RDBMs) are perceived as old and slow. But a well designed and properly setup database can lots of things in realtime (i.e. in a single game frame's time).
@sascha@solidangle databases are used in some places but it’s a tradeoff. Text assets are great in some ways because they’re readable, (somewhat) mergable, and can be searched or even modified without special tools. For having lots of people work in source control it’s also generally better to have lots of smaller individual files that can be modified and checked in independently. It’s also good for content processing pipelines that want to naturally parallelize on multiple input files.
@mjp@sascha@solidangle what's your opinion on implementing the db as a layer on top of whatever format the actual assets use? would the time spent on import/export be too large to justify?
@nicebyte@mjp@sascha@solidangle I’m honestly not sure I have a good grasp of the tradeoffs on that scenario, like I said in the other thread the engine I spent the most time with used optimized and packed arrays for most things for minimal memory footprint and fast iteration. But there was definitely a lineage in that engine coming from single-player games running on consoles with very limited memory. 🙂
@mjp@nicebyte@solidangle That's a good point. I almost forgot we had hardware like the PS3 with 2x256 MBytes, which isn't a lot. But databases are good at compressing stuff and can do normalization (first normal form) and they can be really fast. So you can cram in a lot of stuff even if you only have a few megs.
@mjp@nicebyte@solidangle Decided to have some Sunday programming fun with #Vulkan and #sqlite. Took me less than 30 minutes to build a (very simplistic) sqlite based recording and replay system for my samples. I've been working with real time graphics and database for decades, but never really combined them in such a way. Combining those two very different things was a lot of fun :)
@mjp@solidangle That makes sense. Traditional RDBMs don't play nice with version control (aside from structure), but for delivering content to the user they can be an awesome tool.
@sascha@mjp@solidangle yeah I confess I’ve never worked with or explored such a setup…in my experience the runtime data structures were optimized and packed mostly as plain arrays of data with various mechanisms and concessions for being able to background stream in chunks of the world as you progress. Arrays of various component types are great for blasting through hundreds or thousands of things! But for truly global data a true DB might make sense as a shared resource.
@sascha@solidangle The fear is real. I was on the periphery of the decision process of how the backend should work, and the argument was that SQL is slow and queries can take 30 seconds, so let's use Redis instead. Blizzard uses Redis, so it must be good! (sic!)
Ok, but what is Redis? Oh, it's some sort of "let's keep all data structures in memory" thing. What? It's not disk-backed? How is that supposed to work? Okay, not my circus, not my monkeys, let them handle this.
@solidangle I think WoW and other MMOs use actual real databases for everything which would presumably allow for queries like that. At my previous job there was a system that ran nightly to crawl through all of the content and build a queryable database which was neat. It was mostly used to figure out where assets were referenced though.
@msinilo@mjp@demofox a long time ago my biz partner was looking at replacing the bulk of Unreal's cooker with ULinker metadata dumped into sqllite and running a few queries to get precise lists of assets for cooking, rather than it's behavior at the time of just loading a bunch of stuff to figure out the dependencies.
(I haven't looked to see if the cooker has improved any in UE5, but my experience is each new version of unreal seems to slow down the cooker not speed it up)
@solidangle@msinilo@mjp asset dependency knowledge is an amazing garbage fire in every shippable engine I've seen. It's also a huge problem for memory and disk budgeting, loading in advance to prevent stalls, streaming from the network, and preventing leaks of future content.
@demofox@msinilo@mjp A repeated problem we see with Blueprint-heavy games is spidering content dependencies that load basically the entire game's assets from the root set.
@solidangle@msinilo@mjp that happened in starcraft as well. Like some data record for Terran data specifies what happens when a unit encounters a zerg building. So in a Terran vs Terran match, that gets seen and ends up loading the entire zero asset set. It never went out to the public but yeah... that stuff is really lame. 💯 agree
@demofox@solidangle@msinilo@mjp in Witcher 2 X360 we spent whole year on various memory optimizations (tens of megabytes saved per day, literally), but towards the end, we had a big of game going OOM only if you almost finished it in one session, the worst type of bugs (even speedrun was a few hours for our testers back then).
When a tester caught it, I dumped all the loaded assets, and a dragon (the heaviest asset!) from game prologue was still loaded. Wtf? 1/2
@demofox@solidangle@msinilo@mjp at the previous job all asset dependencies were static for the zone (streamable chunk of the world), with a statically-determined memory footprint. Anything spawned had to have a pre-allocated pool. Very restrictive, obviously! But good for tightening up memory for shipping. 😅
Add comment