As part of my #PhD work, I recently had to perform computation on two very large files using @pandas_dev and I turned to #dask - a set of libraries on top of #pandas, aimed at scaling #python workloads from the laptop to the cluster.
Random #research idea while babysitting a #dask process:
I wonder if there's a way to save a bunch of terms of service documents to be able to version them, and show how they have changed over time, particularly in respect to arbitration, copyright and other #datafication processes?
I'm taking my first foray into #dask - have done the tutorial and read what I can in Stack Overflow.
It's definitely a steep learning curve, but it's been very interesting so far.
@holden's excellent book has been very useful so far, and I think the more I work with it, the more I will master the nuances - how to set up the Client scheduler with optimum workers and threads, the optimum partitions etc.
So im almost finished with my first independent implementation of a standard and I want to write up the process bc it was surprisingly challenging and I learned a lot about how to write them.
I was purposefully experimenting with different methods of translation (eg. Adapter classes vs. pure functions in a build pipeline, recursive functions vs. flattening everything) so the code isnt as sleek as it could be. I had planned on this beforehand, but two major things I learned were a) not just isolating special cases, but making specific means to organize them and make them visible, and b) isolating different layers of the standard (eg. schema language is separate from models is separate from I/O) and not backpropagating special cases between layers.
This is also my first project thats fully in the "new style" of python thats basically a typed language with validating classes, and it makes you write differently but uniformly for the better - it's almost self-testing bc if all the classes validate in an end-to-end test then you know that shit is working as intended. Forcing yourself to deal with errors immediately is the way.
Lots more 2 say but anyway we're like 2 days of work away from a fully independent translation of #NWB to #LinkML that uses @pydantic models + #Dask for arrays. Schema extensions are now no-code: just write the schema (in nwb schema lang or linkml) and poof you can use it. Hoping this makes it way easier for tools to integrate with NWB, and my next step will be to put them in a SQL database and triple store so we can yno more easily share and grab smaller pieces of them and index across lots of datasets.
Then, uh, we'll bridge our data archives + notebooks with the fedi for a new kind of scholarly communication....