Have you ever hired a consultant, or been hired as a consultant, to fix a technical problem that turned out to be a non-technical problem? If so, what do you think the early warning signs were?
My (anonymized) stories of being the consultant in this situation are in the replies
The client had an application where latency was critical. Occasionally the latency was enormous. They wanted me to track down why and fix it. I said, oh, of course I can't run tcpdump on the latency-critical production system, I'll have to figure out a different way. They said, that's no problem. It turned out that the org was proud of having few rules and any of several people could log in to production and run anything they wanted at any time. It was politically impossible to stop this.
The client had a large server farm that parallelized jobs for software developers. Occasionally some jobs took way longer than expected. They were sure it was a storage problem. It's true that they bought a lot of very expensive fancy storage to speed up the jobs which was highly non-deterministic in performance at the firmware level. Also any developer could add anything to the crontab on the servers. Changing either of these things was politically impossible.
Some times that it WAS a technical problem: a software company was evaluating new server options. Performance was startling bad on one combination of hardware and OS that was not significantly different from the others. It turned out to be a combination of poor userland memory allocator tuning and lock contention in the kernel.
Another technical problem: a client was running acceptance tests on new hardware and kept getting the wrong answer. That was a combination of a kernel bug and bad memory.
Another technical problem (I think I'm seeing a pattern): a company was having trouble with storage corruption. The hardware vendor insisted it was a software bug. I proved that the pattern of corruption could only be caused by a hardware problem.
From my personal experience as a systems consultant, the pattern is:
Mysterious latency? It's a management problem
Data corruption or performance problem while evaluating new hardware? It's a technical problem
The more general version seems to be that if the client is having trouble involving some component provided by an outside organization, then it is a technical problem that is politically possible to fix
Sometimes companies will only listen to something if they have to pay money to do it.
What's really wild is when companies have to bring in an outside professional meeting facilitator. The facilitator's job was mainly to get the participants to follow the meeting agenda. I had my arm twisted into being a scribe for one of these in which two departments had to meet for joint project. It was clear that both the departments hated each other.
@vaurora@fanf Not directly myself, but I have a friend who has been a contractor for a company before I was there. Its main job was to discuss with two people that refused to do that by themselves.
When I arrived they were not there anymore :p
Add comment