MAY-JUN 2018

Issue link:

Contents of this Issue


Page 35 of 53

36 INTECH MAY/JUNE 2018 WWW.ISA.ORG SPECIAL SECTION an unusual control program configu- ration that was not considered in the software update. l Condition never considered in the software design. These kinds of prob - lems typically occur when you make a change or addition to the system, creating a condition that was never accounted for in the original software design. It is important to review the history of software updates when troubleshooting and search the update notes online for reported issues. There are different troubleshooting philosophies, but these are common ele- ments and a sequence to start the process: Observe and gather information As much as possible, observe and gath- er information, avoiding preconceived notions. The goal is to understand the current reality about what is happen - ing. These are some typical questions: l What are the symptoms? l When did the problem first start? l What else that is related to this control or automation has unusual data or behavior? Ask people working in the area what they observed. What is different now compared to when things were work- ing properly? Software and firmware issues in sys- tems have increased the complexity and potential problems created by system updates that can change the behavior of applications and controllers. In many sys - tems, the information-gathering process should include the history of software and firmware updates. Record the chronology of the prob- lem and changes made to the system, including operating parameters, alarms, and alerts. Chronology is the science of arranging events in their order of occur- rence. It helps provide an understanding of the problem in the context of the en vi- ronment. As you proceed with trouble- shooting a problem, record the chronol- ogy of steps and information gathered in a notebook or electronically with a tablet computer or smartphone. Review the chronology for clues about the problem. Based on my experience, I cannot stress enough the value of making a chronological and data record as you proceed through troubleshooting. Early in my career I was taught by an experi- enced troubleshooter to carry 3x5 cards and record each step and data as I was troubleshooting automation systems. That was one of the most valuable lessons for troubleshooting. Identify root causes Following a logical path of reasoning based on symptoms can solve many problems, but there are other problems that are more difficult. Sometimes after observing symptoms, the root causes are obvious. With more complex control and automation, however, it takes more ef fort to identify root causes. There may be more than one component (i.e., sensor, actuator, relay power supply, or network communications) that is contributing to the problem. Today this is complicated with software, firmware, network con- figuration, and potential cybersecurity problems. Identifying root causes may follow a logical path of reasoning based on the observable symptoms. In my experience many problems fall in this category, but with greater system complexity there are increasingly bigger troubleshooting challenges. Finding the source of the problem is the detective work of troubleshooting. In complex systems where there can be multiple things contributing to a prob - lem, it is advisable to change one thing at a time and check if that solves the problem. Steps taken should be added to your chronology notes. Open mind It pays to keep an open mind and think outside the box. There may be ways to find the problem that are not obvious. Working with machine tool–based con- trols early in my career, there was a per- plexing problem with integrated circuits that overheated on control boards. When I opened the cabinet, the problem would go away. I wrestled with this for quite a while. Then I talked to a very experienced troubleshooter I worked with, and he reached into his bag of tools and pulled out a hairdryer! I created cardboard baf- fles we used to partition off parts of this large circuit board and first heated one half of the board, then the other, to find the area failing under heat. We simply di vided the problem area in half and did the procedure again, continually narrow- ing the focus. Neat method! I had a similar problem a couple of years later with the minicomputer, and I used my troubleshooting hairdryer to iso- late the problem and fix it. Always keep in mind that troubleshooting is finding an abnormal operating condition that can be caused by a wide range of things. The most valuable discipline when troubleshooting systems is not mak- ing assumptions. In another example years later, I was involved in one of the early industrial network installations of DeviceNet on a liquid crystal display production line. Our company provided the control software that was running the production line. Af ter the line was commissioned and running, there was an intermittent prob - lem with the automation that occurred periodically around midnight. Because this was an important new installation, every vendor involved was on site trying to troubleshoot the prob- lem, bringing in sophisticated network analyzers and other equipment. There was not an absolute root cause estab- lished, but suppliers made changes based on hunches to fix the problem. The problem persisted. One night I went in with the plant op- erations person to watch the operation of the system. When the cleaning person was working in the area, default occurred. We quickly found the root cause. The net - work was implemented with cables inter- connected using round screw-type quick connectors. We noticed the cleaning per - son bumped cable connectors when us- ing a broom to sweep underneath the as- sembly line. We found a cable connector that was not tightly screwed together, and this solved the problem! We all made the faulty assumption in the beginning that since this was a new The most valuable discipline when troubleshooting systems is not making assumptions.

Articles in this issue

Links on this page

Archives of this issue

view archives of InTech - MAY-JUN 2018