The CPU intensive part of the job finishes in less than two minutes. It then takes 6-12 additional minutes back on the main thread to handle all the data that those other threads produced.
OK so for each database row I check if the IP address is being used too much. If it is, I skip the row. If not I'm considered a user of that address, and can process the row. Every "processing" should eliminate the row it comes from in future queries, so the next time I search, once something is finished, the skipped rows will come up again. I can't LIMIT the query though, since I don't know how many rows will be skipped.
And to check how much an address is in use is a network round trip, so I queue up addresses to query for that, and batch send the queries, to get a reply for how many times I can use that address. So first I go through all rows to see what addresses are needed, then send that address-use query. Once I get a reply, I go through all the rows again to see which I can get rid of based on that reply, until I've started enough parallel requests to different addresses.
Except I can't predict how many rows I can get rid of in the second query, so the first query has to iterate over every single database row every time just in case. Not only can it not have a LIMIT, I can't stop queueing up address-use requests until the first query stops returning rows.
But... this query is in theory... not a lot of rows. It's just supposed to search for database records with holes, missing fields that could only be gotten by requesting a specific webpage from a given address. In theory that could mean 7 million user profiles who all failed to load properly because I tried to view them before downloading the avatar image and profile page, but when am I going to look at 7 million profiles at once? It should be like, 10 profiles per page or something, and the address-use query is over the local network so it should happen faster than I can click the "next" button to add on 10 more rows. But I can't guarantee that. Ugh...
An intuitive-but-worth-saying bit I would add to this, which he touches on but does not elaborate: In most of our programming, we can actually expect things to proceed in order, basically corresponding to the lines of code in our program.
“SwiftNIO is Apple non-blocking networking library. It can be used to write either client libraries or server frameworks and works on macOS, iOS and Linux.”
This weekend I was frustrated with my debugging, and just not up to digging in and carefully, meticulously analyzing what was happening. So … I took a left turn (at Alburquerque) and decided to explore an older idea to see if it was interesting and/or useful. My challenging debugging was all about network code, for a collaborative, peer to peer sharing thing; more about that effort some other time.
A bit of back story
A number of years ago when I was working with a solar energy manufacturer, I was living and breathing events, APIs, and running very distributed, sometimes over crap network connections, systems. One of the experiments I did (that worked out extremely well) was to enable distributed tracing across the all the software components, collecting and analyzing traces to support integration testing. Distributed tracing, and the now-popular CNCF OpenTelemetry project weren’t a big thing, but they were around – kind of getting started. The folks (Yuri Shkuro, at least) at Uber had released Jaeger, an open-source trace collector with web-based visualization, which was enough to get started. I wrote about that work back in 2019 (that post still gets some recurring traffic from search engines, although it’s pretty dated now and not entirely useful).
We spun up our services, enabled tracing, and ran integration tests on the whole system. After which, we had the traces available for visual review. It was useful enough that we ended up evolving it so that a single developer could stand up most of their pieces locally (with a sufficiently beefy machine), and capture and view the traces locally. That provided a great feedback loop as they could see performance and flows in the system while they were developing fixes, updates and features. I wanted to see, this time with an iOS/macOS focused library, how far I could get trying to replicate that idea (time boxed to the weekend).
The Experiment!
I’ve been loosely following the server-side swift distributed tracing efforts since it started, and it looked pretty clear that I could use it directly. Moritz Lang publishes swift-otel, which is a Swift native, concurrency supported library. With his examples, it was super quick to hack into my test setup. The library is set up to run with service-lifecycle pieces over SwiftNIO, so there’s a pile of dependencies that come in with it. To add to my library, I’d be a little hesitant, but an integration test thing, I’m totally good with that. There were some quirks to using it with XCTest, most of which I hacked around by shoving the tracer setup into a global actor and exposing an idempotent bootstrap call. With that in place, I added explicit traces into my tests, and then started adding more and more, including into my library, and could see the results in a locally running instance of Jaeger (running Jaeger using Docker).
Some Results
The following image is an overview of the traces generated by a single test (testCreate):
https://josephheck.files.wordpress.com/2024/04/trace_overview.pngThe code I’m working with is all pushing events over web sockets, so inside of the individual spans (which are async closures in my test) I’ve dropped in some span events, one of which is shown in detail below:
https://josephheck.files.wordpress.com/2024/04/trace_with_detail.pngIn a lot of respects, this is akin to dropping in os_signposts that you might view in Instruments, but it’s external to Xcode infrastructure. Don’t get me wrong, I love Instruments and what it does – it’s been amazing and really the gold standard in tooling for me for years – but I was curious how far this approach would get me.
Choices and Challenges
Using something like this in production – with live-running iOS or macOS apps – would be another great end-to-end scenario. More so if the infrastructure your app was working from also used tracing. There’s a separate tracing project at CNCF – OpenTelemetry Swift – that looks oriented towards doing just that. I seriously considered using it, but I didn’t see a way to use that package to instrument my library and not bring in the whole pile of dependencies. With the swift-distributed-tracing library, it’s an easy (and small) dependency add – and you only need to take the hit of the extra dependencies when you want to use the tracing.
And I’ll just “casually” mention that if you pair this with server-side swift efforts, the Hummingbird project has support for distributed tracing currently built in. I expect Vapor support isn’t too far off, and it’s a continued focus to add more distributed tracing support for a number of prevalent server-side swift libraries over this coming summer.
See for Yourself (under construction/YMMV/etc)
I’ve tossed up my hack-job of a wrapper for tracing during testing with iOS and macOS – DistributedTracer, if you want to experiment with this kind of thing yourself. Feel free to use it, although if you’re amazed with the results – ALL credit should go to Moritz, the contributors to his package and the contributors to swift-distributed-tracing, since they did the heavy lifting. The swift-otel library itself is undergoing some major API surface changes – so if you go looking, I worked from the current main branch rather than the latest release. Moritz shared with me that while the API was not completely solid yet, this is more of the pattern he wants to expose for an upcoming 1.0 release.
Onward from here
I might push the DistributedTracer package further in the future. I think there’s real potential there, but it is not without pitfalls. Some of the challenges stem from constantly exporting data from an iOS app, so there’s a privacy (and privacy manifest) bit that needs to be seriously considered. There are also challenges with collecting enough data (but not too much), related choices in sampling so that it aligns with traces generated from infrastructure, as well as how to reliably transfer it from device to an endpoint. Nothing that can’t be overcome, but it’s not a small amount of work either.
Weekend hacking complete, I’m calling this a successful experiment. Okay, now back to actually debugging my library…
Here in part 2 we create some systemd philosophers and see what they're contemplating. It's the first time we've had to clarify that fork meant the eating utensil and not the system call!
And finally part 3, where we actually solve the Dining Philosophers Problem, using a variant of the Chandry/Misra "hygiene" solution. All with only UNIX, systemd and some bash scripting!
are there best practices regarding #concurrency in #Go? I have something like an in-memory database that gets accessed by multiple clients in parallel. The RWMutexes drive me crazy. #golang
I can't lock (wait on) mutex inside fiber, because it locks the whole kernel thread, so I've took a condition variable, put it inside atomic box and made a nutex.