OK so for each database row I check if the IP address is being used too much. If it is, I skip the row. If not I'm considered a user of that address, and can process the row. Every "processing" should eliminate the row it comes from in future queries, so the next time I search, once something is finished, the skipped rows will come up again. I can't LIMIT the query though, since I don't know how many rows will be skipped.
And to check how much an address is in use is a network round trip, so I queue up addresses to query for that, and batch send the queries, to get a reply for how many times I can use that address. So first I go through all rows to see what addresses are needed, then send that address-use query. Once I get a reply, I go through all the rows again to see which I can get rid of based on that reply, until I've started enough parallel requests to different addresses.
Except I can't predict how many rows I can get rid of in the second query, so the first query has to iterate over every single database row every time just in case. Not only can it not have a LIMIT, I can't stop queueing up address-use requests until the first query stops returning rows.
But... this query is in theory... not a lot of rows. It's just supposed to search for database records with holes, missing fields that could only be gotten by requesting a specific webpage from a given address. In theory that could mean 7 million user profiles who all failed to load properly because I tried to view them before downloading the avatar image and profile page, but when am I going to look at 7 million profiles at once? It should be like, 10 profiles per page or something, and the address-use query is over the local network so it should happen faster than I can click the "next" button to add on 10 more rows. But I can't guarantee that. Ugh...