My constant experience with Python over the last ~16 years is I start writing it and it's like, wow, this is so easy, this is great! And then I run my program, and invariably it fails with an inscrutable error/exception 4 layers deep in someone else's library, code I know nothing about— in this case, a wav file library claiming that the number 960 is "not a valid shape"— and I realize that maybe I actually hate Python
@mcc I remember when I started being sort of ok with template meta-programming it hit me that the awful error messages people complained in C++ were just a compile-time form of the runtime error stack traces I'd see in Python, both of which arose from designing code around duck typing.
I used to complain if people didn't check common arg types for null at the top of library functions in Java, and thus found it bizarre that everyone was OK with libraries in Python working this way.
@mcc At least you understand what you're doing. I had barely started reading about Python and it fells like a foreign language. And I can switch from Spanish to English in a flash.
@mcc I stand by my assertion that any Python program longer than a page or three should be rewritten in something more statically/ strongly typed. Apologists will say "proper tooling/IDE support mitigates this" but I reject the notion that good language design requires more than a text editor for effective use.
@mcc It's what you get with a language many/most of whose users are not primarily software people. JavaScript is the same. Libraries are of very varying quality and even the good ones change all the time, meaning there's a huge maintenance overhead.
It's great that languages are approachable by non-specialists, but...
@mcc python is sadly also a headache to embed as a scripting language, precisely because of the universe of useful libraries and the dependency nightmares that brings
I like using python a lot, with numpy, scipy, matplotlib, scikit etc. but it's sadly not very portable at all
@mcc from the other end, super used to Python, have worked in it a long time, but whenever I mess around in a language with static types and good tooling I'm always surprised when the time from first build to first successful run is so brief
This is definitely a problem that typing would have prevented, because if something was wrong with the data I was passing in, it would have been caught at the library boundary rather than deep inside library code.
Gradual typing is something else I want to love. But TypeScript/Typed Python make me think that maybe the appropriate approach is for typing to be optional within a program but mandatory at all module/library boundaries.
@mcc i'm getting more and more behind the idea that "gradual typing" should mean you gradually filling in the types the compiled inferred wrong and thus failed to compile 😅 your point is a perfect enabler, as the library frontier would have well-defined types that percolate through the program code 😊
@mcc Maybe TypeScript works like this (not sure), but in good ol' ActionScript (and later HaXe) it's got gradual typing but having a type is the default. So, you need to specify a type, but it can be */Dynamic. (Possibly "any" in TypeScript?)
It makes the default typing static, and while it's easy to break out of it where applicable that's an explicit choice.
@SonnyBonds So the thing with both TypeScript and Python is they start with a language with no types, and then they added types after a large ecosystem of libraries was written.
I don't think your suggestion necessarily fixes the problem. I guess what you're proposing is in such a world the libraries I consume would simply not contain typing errors, because they'd have been pushed to "usually" use types. I don't trust library authors that much.
@SonnyBonds As for TypeScript specifically, it's relatively strict about where it requires types by default, but it doesn't matter, because you can simply write an entire file in JavaScript and then it's fully untyped.
@mcc Yeah sorry I didn't mean it as a fix for existing systems. I just mean it would've been a better choice from the start, since it encourages better type use. That ship has sailed.
Moving from general thoughts to specific— I thought this would turn out to be me making an error with the data type I passed in. Except no. I checked and I defs put in a regular Python array of length 960. The (960,) 1-tuple is just the dimension of my array, and the "Invalid shape" appears to be something emitted from Numpy, which this wav library inexplicably uses. So I'm gonna have to learn Numpy to debug this library to write a file. Nightmarish. It might be quicker to write my own wav lib
@delta_vee In many languages, it's certainly part of the type, yes. And at the external interface boundary with which I am interacting with this library, there are no types in scope which are dynamic array types. I am passing in regular Python arrays, for which shape (dimension, if not size) is absolutely visible, mandatorily, to the type system.
So after reading the source of this library, it turns out that "Invalid shape: (960,)" means "I was passing in one array of channel-interleaved sound samples, but it wanted an array of two arrays". Which in my opinion is not a clear error message since instead of describing the error in terms of the semantics of the library (stereo data expected, mono data received) it described it in terms of some dynamic-dimension scientific computing array type it's using internally
Okay. I think this library was simply a honeypot. After understanding what format it wants the data in, and making a list of 3-4 bugs to file ranging everything from "unhelpful error messages" to "your pip dependencies are incorrect" to "the link to your current documentation is dead", I open the file I made and the wav file consists entirely of zeroes. The input, by observation using "print", is not zeroes. The library apparently does nothing at all.
Rubbing eyes Alright. So once it became clear that (although it advertises use with or without numpy) this library does not work without numpy, I decided to throw good time after bad and use numpy. The library now works.
I do have a numpy question now.
I build my array of data gradually. In my working code, I build a 2d python array [[],[]], then pass it to np.array.
Is there a way to gradually build/append an np array?
Do I have to first preallocate a size with zeroes() then use put()?
@Scmbradley I just feel like I'm creating so much garbage. I thought the entire point of numpy was it let you do arrays without allocating lots of temporaries.
@mcc np.append appends to a copy of the array given, i believe. so it's a new allocation, but it does give you what you want. i haven't deeply investigated the perf impact.
@jplebreton OK. So what I've got is a 2xVerylong matrix, and I am annoyed that constructing the matrix means making a Verylong number of 2-length arrays as garbage. So if the pure-numpy doing this means creating lots of numpy temporaries rather than lots of array temporaries it doesn't sound like a win.
@mcc it seems especially ill-suited for generating a series of WAV data! in my experience (esp writing Playscii) numpy gets used everywhere because it's the powerful swiss army knife (whose native parts seem to have pretty good performance, IME) but it ends up having a not-small impact on all the code that refers to it, and sometimes the ideal paradigms for numpy VS the client project don't match particularly well.
@mcc i'd be curious how people handle this kind of thing in languages that require you to think more about allocation, eg Rust - there's got to be some pattern that works well for "i'm recording sound and i don't know how long it's gonna be" (allocate in blocks as you go?) and i'd really hope numpy supports that.
@mcc@jplebreton If you can’t predict the total array length, would it be possible to generate the data in chunks? You could build a list of NumPy arrays and then concatenate them into a single NumPy array at the end.
This would avoid creating a list of 2-element sample lists or appending one sample at a time to the NumPy array (which would be slow if each append reallocates and copies the entire array).
@mcc@jplebreton rather than create python (nested) lists and convert them with numpy.array(), you can avoid garbage by creating a numpy array with numpy.empty() or numpy.zeros(). you can fill it by indexing or slicing, or from an iterable.
additionally, numpy treats array layout as distinct from storage; so you can create a long 1d numpy array of left,right,left,right samples, then set the array shape to (-1,2) so it is viewed the same as if it was an array of 2-element-tuples.
@mcc@jplebreton What I’d do here is create a [VeryLongList, VeryLongList], then use np.asarray (and take the transpose of the result if you need the opposite ordering); this should create less garbage than using np.append. numpy is heavily optimised for doing operations on arrays that stay the same size. Preallocating (in chunks or otherwise) is another option as discussed elsewhere in the thread, and I think would have a lower memory overhead if that’s a concern.
@mcc growing a numpy array can be done but it requires allocating a new one and copying the old one into it so yes allocate the size you need with zeros().
@mcc The docstrings for this library indicate that it uses NumPy types in its public interface, so I think the authors expect you to learn NumPy in order to use it at all.
I would try reading a WAV file and seeing what array shape it returns. Maybe you need to transpose rows and columns?
@mcc My best story about the importance of typing was a piece of code I wrote several years ago to help me manage a largish course. There was a bug in part of the code that paired students for group projects. I couldn't be bothered to hunt down the bug because I was the only person using it, and the erroneous output was easy enough to catch.
One summer as I was adding functionality, I decided to also add type inferences. Immediately my IDE said "Hey, you've got a bug here. You're assigning a Duck object to a Goose variable." (not the real class names)
Add comment