This is the final in a series of 4 blogposts on this topic of troubleshooting. The first 3 parts can be found here, here and here
At the risk of sounding like a debater, let me begin with a definition: The Oxford English Dictionary defines Analysis as
Detailed examination of the elements or structure of something
Another good word for this step would the Theorising. OED gives us:
Form a theory or theories about something
Basically we look at the information we have gathered and try to come up with an explanation for the occurrence from it. The order is most important as if you theorise before getting some data the temptation is to make the data fit the theory rather than the theory fit the data. Holmes was very clear on the unacceptability of this in the previous post! Most of this phase changes from issue to issue and there are a lot of times where this phase is done in parallel with the data collection phase (sometimes the cause is obvious when you get some information on the problem). While there is much variation in how the analysis proceeds there are a few general questions that always need to be answered:
- Patch levels: is the product on a supported version?
- Does the issue description match a known bug?
- Check event logs around the time of the known occurrences of the issue. Any items of note should be recorded even if they don’t appear relevant at the moment, may be useful later
- Confirm it’s not a configuration issue.
The final phase – and if you’re working in a customer facing support role, the most important one – is giving results to the interested parties.
The end result of the analysis will take one of four forms:
- The cause is identified and the issue resolved
- The cause is not identified but an acceptable work around is found
- The cause is not identified and no workaround is possible. Needs to be escalated to vendor
- The cause is not identified and you have to refine the approach and find more information
Of the above results I would honestly only consider #1 and #3 to be the end of the matter. With 1 you have resolved the problem and with 3 its escalated to the vendor and you have to work with them but with specialist help a solution should be found or at least a cause identified. I would consider #2 to be ok in the short to medium term but not a valid solution in the long term and obviously #4 means you have to start the whole process again.
Hopefully the time spent reading these posts will help refine whatever method you use to troubleshoot issues.
This is the 3rd post in a series of 4. Parts 1 and 2 are available here and here
For a completely fictional character, Sherlock Holmes does give some great quotes
“It is a capital mistake to theorize before one has data. Insensibly, one begins to twist facts to suit theories, instead of theories to suit facts.”
To solve any problem you need data on what is happening. Without data, you are effectively playing darts blindfolded. Data takes many forms and it’s not all about log collection. In all honesty, logs are usually one of the last things you look for. Unless you have some idea of what you are looking for, there is just too much information to parse. The only reasonably common exception to this would be the Windows event logs as (a) they really don’t record verbose logs and (b) if you have timings for when the problem occurred you can really focus your searches. Generally though the answers from the scoping questions (listed in previous post) are much more useful during the initial phase of the troubleshooting. Of the scoping questions the most important questions are:
- Has this ever worked?
- Is it a supported configuration?
If the configuration is never going to work you are trying to perform a miracle rather than troubleshoot, and miracles in production environments are best avoided.
It’s hard to give specific recommendations about what data to collect as this will vary massively from problem to problem. In fact in my experience the scoping questions usually end up being the data collection for all but the most intractable of problems. While I can’t give specific recommendations about information required there is some general information always required:
- Environmental information ( Virtualisation, physical, Operating system, etc.)
- Software versions and hotfix/patch level
- For Windows servers, Event logs are always a good place to start
- Reproducibility or timings of when the issue occurs/occurred
- Screenshots of error messages or videos of the event happening.
That last one may seem strange but that whole “picture being a thousand words ” isn’t a total exaggeration. In a previous job with a software vendor I had a case once where a customer was having a graphical display problem with the software. After sometime troubleshooting it for some time and getting nowhere I eventually showed the video the customer had sent me to a colleague who knew exactly what bug it was and supplied me with a fix. This illustrates two things: 1) don’t be afraid to ask for help and 2) pictures/videos can be the bit you need to solve the problem
Part 1 is available here
The following Einstein quote is probably apocryphal, but that doesn’t make it any less useful:
“If I had an hour to solve a problem and my life depended on the solution, I would spend the first 55 minutes determining the proper question to ask, for once I know the proper question, I could solve the problem in less than five minutes.”
Scoping the problem or determining the proper question to ask is honestly the most important part of the whole troubleshooting process, as if you don’t know what the problem is how can you fix it? Also, if you don’t define the problem how do you know it’s a problem? That one is a little odd but true but the “issue” could just be a configuration that is never going to work.
A long time ago I used to be broadband support tech in a ISP. Wireless routers were new on the market and just being rolled out. Soon after I started supporting broadband (previously I was supporting only dial-up) I had a call with a customer who couldn’t get their internet working. After about 5 minutes struggling to get a handle on the problem and checking some things on the computer, I went back to the beginning and asked them where their router was. The response was, “Oh that thing? That’s in the shed. Why would I need it – the internet is wireless.”
This one has always stuck in my head. If I had have nailed down what the issue was in the beginning, it would have been a faster solution and there would have been far less messing around. It also illustrates that sometimes the issue may not be a technical problem but a configuration problem. The setup the customer had was obviously never going to work. Correct scoping helps fix these too, and helps keep you clear of rabbit holes.
For scoping a problem I generally have a list of questions that I run through. Some, or even all of them, may not apply to every case, but it’s a good starting point.
- What is happening? By this I don’t mean the overall issue, I mean the symptoms of the issue. Computer blue screening would be an obvious example of a symptom rather than an issue.
- Has this setup ever worked? Is it a new configuration or something that has been in place for some time?
- What is the configuration trying to accomplish?
- If it has, when was the last time it worked?
- How many issues are they experiencing? If more than one, are they all different or related?
- How many users is this affecting? How many servers is this affecting?
- How severe is the problem? Crashes , slowness, application crashes or just a vague feeling of unease.
- If intermittent, how frequent or random is it?
- Is it reproducible?
Like a lot of people working in IT, I ended up doing this job because of the enjoyment I get out of fixing things and generally tinkering with various bits of technology. Curiosity about how things work would probably be the biggest driver in my career (and life – but that is another topic!) Thinking back on it, I would have to say that my earliest memory of troubleshooting would be “helping” fix the upstairs phone with my father (who is a telecoms engineer) when I was about 5. This was a great introduction to troubleshooting as it was such a hands-on problem, but more on this later.
Over the years I’ve spent working in technical support, I’ve noticed people tend to troubleshoot things in one of two ways:
- Working on intuition and essentially just randomly making changes with no real reason for making the changes (The Potluck Approach)
- Gathering some information, drawing some conclusions and then making changes (The Structured Approach)
The second method is rarer than you would think. Sometimes randomly pushing buttons will fix a problem faster, simply through luck, but overall taking a structured approach is faster and results in fewer disasters caused by pushing the wrong button. It also means you generally learn what caused the problem and thus can prevent it from happening again.
My approach to problems is straightforward enough but I’ve found it does help. Below is my general, step by step, approach:
- Scoping the problem: What is happening and (sometimes) why is this a problem?
- Data collection: Varies from problem to problem but usually includes environmental information, version of software, when it last worked etc.
- Analysis: Looking at the data collected and seeing if there are any indications of a problem.
- Result and conclusions: This varies based on where the analysis has led you. Sometimes you’ll have to go back, change what you’re looking for, and take a different tack.
The result of the historic phone troubleshooting? We ran the cables and initially it didn’t work. Then we checked the first junction box and got a signal, so the problem was between the junction box and the terminating socket. Turned out it was the socket. This might be a basic example, but the benefit of a structured approach is that it applies to all problems. If we apply the structure to the steps taken, it would look like this:
- Scoped the problem: Phone wasn’t working.
- Data collection: Checked how far the signal was getting.
- Analysis: Problem was between junction box and socket.
- Results: As replacing the socket was easier than ripping the cable out of the wall, we tried replacing that and hey presto it worked without tearing the wall apart. Success!