BluePower

BluePower, 2 months ago

Update regarding on the revisions: The bug seems to affect later revisions, and not the entire revision list. And here’s what I’ve done so far.

When trying to download some earlier revisions in power-generator-manager, I have to say that revisions only made after 12 January 2024, 23:31:11 UTC+7 (after index 863) are affected by this bug. I even ran this line of code in the console to get a decoded URI of the latest model patch:

<span style="color:#0086b3;">decodeURI</span><span style="color:#323232;">(diffStuff._patchesToRevision(diffStuff.modelTextPatches, </span><span style="color:#0086b3;">1617</span><span style="color:#323232;">))
</span>

And that returned the exact same error, URI malformed. So that’s a problem with the encoded text on each specific revision.

But my investigation doesn’t stop there. To see if there’s a single character that triggers the entire URI corruption, I also compared those raw revision data of index 863 and 864, decoded manually, in a text comparison app, and I’ve finally found the culprit, in line 204:

Revision Sat Jan 13 2024 13:09:52 GMT+0700 (GMT+07:00):

<span style="color:#323232;"><</span><span style="color:#63a35c;">li</span><span style="color:#323232;">><</span><span style="color:#63a35c;">b</span><span style="color:#323232;">>The generator manager is getting another big update!</</span><span style="color:#63a35c;">b</span><span style="color:#323232;">> ... Scroll down past the end of the announcement to see the full changelog of the update!</</span><span style="color:#63a35c;">li</span><span style="color:#323232;">>
</span>

Revision Fri Jan 12 2024 23:31:11 GMT+0700 (GMT+07:00):

<span style="color:#323232;"><</span><span style="color:#63a35c;">li</span><span style="color:#323232;">><</span><span style="color:#63a35c;">b</span><span style="color:#323232;">>The generator manager is getting another big update!</</span><span style="color:#63a35c;">b</span><span style="color:#323232;">> ... Scroll down past20the announcement to see the full changelog of the%2!</</span><span style="color:#63a35c;">li</span><span style="color:#323232;">>
</span>

And there it is, the %2 that triggered the URI malformed problem. There was apparently some sort of a glitch with the diffStuff._patchesToRevision() function when it tries to parse the raw patch data into a revision (that could be on the concurrent save requests you’ve mentioned).

I then tried the same procedure in the latest revision of the hub page and found that the %2 character started spreading around the generator code multiple times. Not only that, but parts of the entire data are also scrambled pretty bad, too. (This doesn’t happen on unaffected revisions!) So your statement on “your current revisions are corrupted” is kind of true.

And now, every time I had to get a revision, I’d have to get the patch directly with that specific approach, manually decode it, transfer it into a notepad, and then saving it if I wanted to (unless there’s a way to fix that URI corruption thing).

And yes, I did check the outputTemplatePatches and everything in it is okay, no corruption whatsoever. Even to the latest one 🙂

BluePower, 2 months ago (edited 2 months ago)

Thanks! I will try to find more generators affected by this bug and maybe start messing around with that blank page (revisions worked on 2 of my easter egg generators so far). But for now, here’s a little snippet that I would use to check if a generator’s revision is affected by the bug:

<span style="font-weight:bold;color:#a71d5d;">for </span><span style="color:#323232;">(</span><span style="font-weight:bold;color:#a71d5d;">let </span><span style="color:#323232;">i </span><span style="font-weight:bold;color:#a71d5d;">= </span><span style="color:#0086b3;">0</span><span style="color:#323232;">; i </span><span style="font-weight:bold;color:#a71d5d;">< </span><span style="color:#323232;">diffStuff.modelTextPatches.length; i</span><span style="font-weight:bold;color:#a71d5d;">++</span><span style="color:#323232;">) {
</span><span style="color:#323232;">  </span><span style="color:#0086b3;">decodeURI</span><span style="color:#323232;">(diffStuff._patchesToRevision(diffStuff.modelTextPatches, i));
</span><span style="color:#323232;">  </span><span style="font-style:italic;color:#969896;">// when executed, the console tries to evaluate that so that's why it'll get an error if the revision's corrupted
</span><span style="color:#323232;">  </span><span style="color:#795da3;">console</span><span style="color:#323232;">.</span><span style="color:#0086b3;">log</span><span style="color:#323232;">(</span><span style="color:#183691;">"modelText: " </span><span style="font-weight:bold;color:#a71d5d;">+ </span><span style="color:#323232;">i);
</span><span style="color:#323232;">  </span><span style="font-style:italic;color:#969896;">// count COUNT COUNT
</span><span style="color:#323232;">}
</span><span style="font-weight:bold;color:#a71d5d;">for </span><span style="color:#323232;">(</span><span style="font-weight:bold;color:#a71d5d;">let </span><span style="color:#323232;">i </span><span style="font-weight:bold;color:#a71d5d;">= </span><span style="color:#0086b3;">0</span><span style="color:#323232;">; i </span><span style="font-weight:bold;color:#a71d5d;">< </span><span style="color:#323232;">diffStuff.outputTemplatePatches.length; i</span><span style="font-weight:bold;color:#a71d5d;">++</span><span style="color:#323232;">) {
</span><span style="color:#323232;">  </span><span style="color:#0086b3;">decodeURI</span><span style="color:#323232;">(diffStuff._patchesToRevision(diffStuff.outputTemplatePatches, i));
</span><span style="color:#323232;">  </span><span style="font-style:italic;color:#969896;">// same with this too ^^
</span><span style="color:#323232;">  </span><span style="color:#795da3;">console</span><span style="color:#323232;">.</span><span style="color:#0086b3;">log</span><span style="color:#323232;">(</span><span style="color:#183691;">"outputTemp: " </span><span style="font-weight:bold;color:#a71d5d;">+ </span><span style="color:#323232;">i);
</span><span style="color:#323232;">}
</span><span style="font-style:italic;color:#969896;">// you'd have to load the revisions in order for it to work, and for generators with lots of revisions, it'll probably take longer to finish the check
</span>

Which in this case, I just ran on my milestone announcement page, and it returned this:

https://sh.itjust.works/pictrs/image/815bb502-6fb7-4958-ba43-dece51aa108c.png

(Also @VioneT, @Alllo, @eatham and others, if you wanted to contribute by finding some buggy revisions in your generators by running that in the console and found errors like that, that’ll be very helpful.)

Why not fork it and start with a fresh? That will reset your revisions.

Seems like you have a good point. 🙂 I will try this with my milestone announcement page after my preparation with it is fully done.

BluePower, 2 months ago

Nice! At least you’re trying to :)

BluePower, 2 months ago (edited 2 months ago)

That’s interesting. I didn’t know the hub also has the same revisions problem. I’ve been tinkering with that code for hours and this is what I made:

<span style="color:#323232;">function checkForMalformedRevs(patch, loglabel, startFrom) {
</span><span style="color:#323232;">  let text;
</span><span style="color:#323232;">  let dateRegex = [/^.* ([0-9][0-9]:[0-9][0-9]:[0-9][0-9]) GMT+.*/g, "$1"];
</span><span style="color:#323232;">  console.log('[' + loglabel + ']', Date().replace(/ (GMT+.*)/g, ""), 'Check Started.');
</span><span style="color:#323232;">  patch.forEach((a,i) => {
</span><span style="color:#323232;">    if (i < startFrom) return;
</span><span style="color:#323232;">    let pi = diffStuff._patchesToRevision(patch, i);
</span><span style="color:#323232;">    let x = 0;
</span><span style="color:#323232;">    let offset = 0;
</span><span style="color:#323232;">    try {
</span><span style="color:#323232;">      for (x = 0; x < pi.length; x += 100) {
</span><span style="color:#323232;">        offset = (!!pi.substring(x, x + 100).match(/%.$/g) * 1) + (!!pi.substring(x, x + 100).match(/%$/g) * 2); // checks if there's any "chopped" URI characters at the end of the chunk to prevent `URI malformed` errors just because of that
</span><span style="color:#323232;">        text = decodeURI(pi.substring(x, x + 100 + offset));
</span><span style="color:#323232;">        text = '';
</span><span style="color:#323232;">        x += offset;
</span><span style="color:#323232;">        offset = 0;
</span><span style="color:#323232;">      }
</span><span style="color:#323232;">      console.log('[' + loglabel + ']', Date().replace(...dateRegex), 'Checked:', i);
</span><span style="color:#323232;">      x = 0;
</span><span style="color:#323232;">    } catch (err) {
</span><span style="color:#323232;">      console.error('[' + loglabel + ']', Date().replace(...dateRegex), 'Error:', i, 'at chunk', x + '-' + (x + 100 + offset), 'of', pi.length, err);
</span><span style="color:#323232;">    }
</span><span style="color:#323232;">  })
</span><span style="color:#323232;">  console.log('[' + loglabel + ']', Date().replace(/ (GMT+.*)/g, ""), 'Check Complete.');
</span><span style="color:#323232;">}
</span><span style="color:#323232;">checkForMalformedRevs(diffStuff.modelTextPatches, "modelTextPatches");
</span><span style="color:#323232;">checkForMalformedRevs(diffStuff.outputTemplatePatches, "outputTemplatePatches");
</span>

There we go! It’s a greatly enhanced version of the checker tool. To reduce all the wait, the checker first processes a chunk of data, then if that chunk of data has been checked successfully, then the next chunk will be checked. (But still, if the revision data has around tens of thousands of characters it’s going to take seconds to process)

The process roughly looks like this:

<span style="color:#323232;">"generatorStats%20=%20%7Bimport:generator-stats-plugin%7D%0%0Atitle%0A%20%20Power%20Generator%20Manager%0A%0A%0A%0Ann"
</span><span style="color:#323232;">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ < checking
</span>

The checker will take a part of the data and then decodeURI’s them. Then if it’s successful, another chunk is also checked:

<span style="color:#323232;">"generatorStats%20=%20%7Bimport:generator-stats-plugin%7D%0%0Atitle%0A%20%20Power%20Generator%20Manager%0A%0A%0A%0Ann"
</span><span style="color:#323232;">                                                  ~~~~~~~~!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ < checking
</span>

And there, once it’s URI malformed time, the checker stops for checking further chunks. Additionally, the logger not only throws the error, but also locates the chunk location where the error is throwed and shows when the check happened in time. You can also set on which index the checker should start checking with startFrom for easier debugging.

Additional note: While testing, I also found that other “suspicious” (invalid) characters (including %9F) and the single percentage % are also affecting this whole revisions problem.

(Also I might create a whole “utility” generator out of this so more people can contribute to this 😊)

BluePower, 2 months ago

What? With a hub that had over a thousand views? That’s remarkable.

Also @perchance, if there’s a way to backup and erase all the revisions somehow 🤔

BluePower, 2 months ago

Oooh nice! I’ll test that later, and I’ll improve it even more when needed.

BluePower, 2 months ago (edited 2 months ago)

That’s what I’m talking about. I could set my own word filters, and everyone else can set their own word filters as well, and they can be different for each user. Kind of like a combination of bannedWords and the user blocking feature. 🙂

I’m not as sure how it would affect your conversations since if they messaged with a filtered word, you might not have the correct context to reply to.

Also, for this, there should be a command to show/hide messages containing the word filter (that marked with a specific flag). It’s like the suggested “unblock all” command for words I previously mentioned in the post but the defined blocked words do not reset. With that, I’ll be able to look up and still see those replies with the word filters.

BluePower, 2 months ago

Kind of the same thing happened to me when commenting on my Popularity Achievement generator. It loses the admin tag when I refreshed the comments frame.

https://sh.itjust.works/pictrs/image/3dc59d7f-69de-4486-b299-baa3f94d158c.png