I just want to run some spike detection code I made a while ago.
Instead of spikes I get a weird error. So now I need to update package 1, which requires updating package 2, 3, 4, 7, and 28, which in turn want a newer version of python (except package 9 which refuses to work now of course), so I also need to reinstall anaconda completely (fuck knows why the upgrade button never works)...
And of course none of that actually runs, so I need to figure out how to make things go in a docker container that is in turn wrapped in whatever the hell a singularity is?
@susanleemburg
Totally hear you. This is why I dont use them and suggest others dont as well - once one piece of something gets locked into using a container or virtualization environment then the rest of it has to too, and it get unnecessarily complicated quick.
@jonny@susanleemburg are containers the same as virtual environments in this context? @susanleemburg I feel you. I’ve been adapting some spikeinterface code and it’s been a real hassle. I call it “the Linux spirit” 😅 (and I used to be a Python defender but once you get used to how easy Matlab is and how it (mostly) “just works” it’s hard to go back.)
@elduvelle @susanleemburg
Virtual environments are different bc they are part of how python works, conda/docker/etc. Are "on top" of that and end up ratcheting up complexity. Ill hold my tongue on MATLAB bc I think the "just works" is a short term gain and a long term huge loss
@jonny@elduvelle@susanleemburg I had the opposite feeling, that everything with Matlab for me was always a struggle and that it was a joy to move over to Python for that reason. However, it's definitely the case that the Python ecosystem has gotten worse in terms of dependency hell than it used to be. I do wonder if the easy availability of virtual environments and docker etc. have made it easier to be lazy about backwards compatibility when developing packages. We test @briansimulator on a huge array of combinations of different versions of python, operating system, etc. to guard against this and it's not that difficult to set up a continuous integration infrastructure to do this using GitHub actions. I wish more people would.
@neuralreckoning@elduvelle@susanleemburg@briansimulator ya it's gotten a lot easier to spec dependencies and so if you don't do that (as in the case of a private use script, which is normal) then you don't get access to all the dep resolution stuff that has been made in the last few years. so i just make everything a package now since it's so easy, but that doesn't really help if you haven't done that in the past.
@jonny@neuralreckoning@elduvelle@briansimulator I'm much more at home in matlab than in python, so it's very likely that I'm a pretty big factor in whatever mess my python scripts end up being. But I'd like to not have to wonder if a year-old script still works, or if something has become not-quite compatible in the mean time. Particularly this morning when I had planned to to many other things instead. Sigh.
I guess my mess was the flipside of having the version-flexibility/specificity in python that you don't have in matlab (at least not in the same way).
I've never really gotten along with docker, so I mostly just run different anaconda environments for now. I refuse to figure out singularity out of spite, for the moment.
@neuralreckoning @susanleemburg@elduvelle@briansimulator
The way you would make this easy is to make your scripts into a package (using something like pdm you can just do pdm init) and then specify your dependencies (pdm add). Then it does "just work" because that generates a lockfile with all the versions correctly resolved that you can go back to forever. Making everything backwards compatible forever means packages cant evolve, which is one of the strengths of Python's package ecosystem vs. Static MATLAB builtins, and the alternative is to spec your dependencies which is more or less a single command nowadays.
It might seem finnicky but the intuition is basically "I have code. My code needs other code. I describe what code my code needs" - so when writing a script and you need a package, you just do pdm add package and it both handles adding that dependency and installing it in your venv. P seamless once you get used to it, but ya if you didnt do that in the past then you need to do it now which is a pain when as u say u have other things to do.
The mirror situation in MATLAB is getting a script that uses a bunch of functions that are defined ~ somewhere ~ in the PATH of the person that wrote it, but they might have only been present on that one machine that one time since MATLAB doesnt have a packaging system per se, so the appearance of "just working" in MATLAB is essentially at the expense of having any code that isnt builtin, in one of the extension modules, or hard copied and pasted into the thing youre distributing. A steep tradeoff indeed, since the depth of tool composition becomes exactly 1.
@jonny@susanleemburg@elduvelle@briansimulator I don't agree that enforcing backwards compatibility is a bad idea. Otherwise you get the problem that package X requires a particular range of versions of Y, but package Z requires a non overlapping range of versions of Y meaning that X and Z can't be used together. Python ecosystem rife with this sort of problem. Backwards compatibility eliminates this problem. Just upgrade everything.
@neuralreckoning @susanleemburg@elduvelle@briansimulator
Well yeah im not saying backwards compatibility is bad, but in some cases its impossible to maintain forever. Ample deprecation warnings and version range pinning addresses the problem in a way that balances dynamism of ecosystem with stability. See all CRAN or CTAN as an example that is more focused on stability at the expense of dynamism.
@neuralreckoning
One recent example is pydantic 1->2. That broke a ton of packages, but the changes were uniformly good and made everything work better after the change was swallowed. Like it would have been better to get way more advance notice but just to say there are times when breaking backwards compat is necessary and good. It does require maintenance, but its still possible to use all those old versions with correct upper bounds in their dependencies. For people who dont want to worry about all this and dont intend their code to be used by other packages, doing exact version pins works great and achieves immortal working code.
@jonny I'll agree with there being times when it's necessary. We broke compatibility once with Brian in it's 17 years so far, and that seems like a reasonable frequency. But we also made a new package name so actually no breaks in compatibility. My worry is when breaks are frequent enough that you can't install two packages because of non overlapping requirements. This happened to me more than once, so it's a real problem.
@jonny@elduvelle I actually don't have a huge preference for matlab over python because of the software itself. It's really more based on my own skill.
The newer matlabs have a number of features that I absolutely despise (being able to edit the data directly without coding has to be some of the worst).
Add comment