what force might have coerced Microsoft to behave more reasonably, in that situation?
Strong antitrust and anti-corruption laws. Their actions were not “unreasonable”, they were straight up illegal.
Edit: also you should read up on the whole thing. They didn’t break compatibility with their own office suite of course. What they did is lie to (and almost definitely pay off) the standardization body: “here is the spec for OpenXML, you see we’re open it’s right here in the name, anyone can implement it and be interoperable with us”. So OpenXML was standardized along with OpenOffice’s OOXML (at the start of the process, only OOXML was considered for standardization).
Once the deed was done, they of course didn’t implement OOXML in MS Office (as is their right), but they also didn’t implement their own OpenXML spec properly, which means OpenOffice still had to reverse-engineer an intentionally obfuscated and broken format to try and read/write documents compatible with MSO.
So the whole thing has been absolutely useless, except for a couple of “experts” from the panel who came out of it a bit richer.