82nd General Meeting Speaker Presentation
Inspection Quality: One AIA’s Perspective
By Fred Bull
The following presentation was delivered at the 82nd General Meeting on Monday, May 13, by Fred Bull. It has been edited for content and phrasing. To follow along with Mr. Bull’s slide presentation, click here.
For the past eight years, Fred Bull has served as the president and chief executive officer of HSB Global Standards. HSB Global Standards is an authorized inspection agency (AIA) that provides global pressure equipment conformity assessment services. Mr. Bull has a degree in electrical engineering from the University of Virginia. He joined HSB Global Standards in 2002. Prior to joining HSB, he spent 23 years with General Electric in a variety of engineering, technical sales and management positions. These positions primarily focused on industrial control systems and electrical power distribution. Mr. Bull's presentation this afternoon is entitled Inspection Quality: One AIA's Perspective.
The National Board of Boiler and Pressure Vessel Inspectors was formed in 1919 with the objective of promoting greater safety for life and property through the uniform application of construction, repair, maintenance, and inspection of pressure equipment. As the president of the largest employer of National Board commissioned inspectors, with over 750,000 hours of inspections performed a year, I am here to present an overview of the challenges and opportunities to meet the objective of the National Board through uniformity in construction and inspection of pressure equipment.
First, I would like to provide some context. In August 2002, I had 23 years of very diverse experience providing electrical systems to commercial, industrial, and utility customers who are also large users of pressure equipment. At that time I never heard of Hartford Steam Boiler, the National Board, the ASME boiler code, or property and casualty insurers. I knew generators made electricity, but I had never heard of a steam generator. In October 2002, I was hired as director of marketing and sales for HSB Global Standards, and in 2004, I was promoted to my current position. As I began learning about my new employer, and specifically, about the authorized inspection agency business, I was astonished.
Although I had joined a very successful company with an excellent, long-standing reputation, I had never seen a business model that included the following: first, my service delivery staff, the inspectors, were certified by an outside organization with minimal requirements to maintain their certification. Second, the inspectors self-dispatched. They decided when they would go, where they would go, and how much inspection they would do based on their interpretation of the code. An accreditation was required of us, but we only had to do it for three days every three years, and ASME did the accreditation.
I was looking for metrics, but the only industry-wide metrics I could find were the National Board's statistics reports on violations. The primary assessment of quality was our company's and the market's qualitative assessment of our reputation. I didn't believe this model could possibly work. So in order to get a customer's perspective, I went out and visited a number of customers. They all told me HSB did a great job, they loved their inspectors, and they had no idea why I would be spending my time coming out to ask them these silly questions.
So the process worked, but I had no idea why. Upon further investigation, I found that from around 1920, the underlying, unstated assumptions that held all of this together was ASME, which developed and maintained the code; insurers, who bore the risk of loss; and jurisdictions, which had the responsibility to protect the public. Basically, everyone worked together due to common interest, with a substantial amount of work relying on common practices and institutional knowledge passed down through generations.
While this approach has been successful, the unstated, underlying assumptions have changed significantly. One of the original assumptions was that insurers bear the risk of loss. In 1920, insurers inspected the equipment under construction; they had insured the boilers they inspected, so there was a direct correlation between the risk, the loss, and the inspection.
Today, very few insurers do shop inspection of construction on vessels they insure. Another assumption was that owners, insurers, jurisdictions, and manufacturers all understood the ASME code in the same way and most of the work was executed in North America. Today, most pressure equipment is manufactured globally for global owners. It's entirely possible that a boiler going into your jurisdiction or to the state of Florida was manufactured by a brand – a shop with their first boiler; with a brand new AI; with a brand new supervisor; with an authorized inspection agency that has very little experience with ASME.
Another assumption was that inspectors are an integral part of insurance companies. AIAs are no longer part of insurance companies (or no longer required to be insurance companies) but the underlying assumptions about AIAs still exist. Previously, the AI spent most of his time working with the boiler and various ASME codes. Today many AIs, especially overseas, may spend less than 10 percent of their time working on ASME.
Insurance companies tend to view inspections as risk-reduction activities rather than profit-making activities. Therefore, the insurance company has a vested interest in a high level of training. The insurance company also has a vested interest in making sure that sufficient inspection is done to reduce the risk. As companies that do inspections for profit, inspection companies are often focused on productivity, which may include reducing the hours to accomplish a task or reducing costs by providing minimal training and technical support.
Another assumption was that QAI-1 was written to give the AI flexibility based on his employer, which was the insurance company, flexibility so that they could reduce the risk. But this flexibility could also be used to reduce inspection hours and increase productivity. So to this point, these are all the unstated underlying assumptions.
So let me get to the point of my presentation, which is the quality of inspections for construction. Today's method of assuring quality of inspections by the AIA is to address inspection deficiencies as they occur (essentially, a 1920s approach to quality). A modern approach of assessing the quality of AIA inspections would have quality metrics on key performance indicators and would be consistently applied and audited across all AIAs. With AIAs changing from insurance companies to inspection companies and with many AIs doing ASME work part-time, this becomes even more important.
Now, you may not think there has been enough change to these underlying assumptions to warrant a significant amount of effort, but let me show you some data. This is the profile of the 15 current authorized inspection agencies excluding jurisdictions that hold certificates of accreditation. Most of the six AIAs at the bottom of this chart are traditional insurers with over two billion dollars of direct written premium and do virtually no code inspections on new construction.
The middle four AIAs are essentially inspection companies with an insurance entity that they needed to comply with earlier requirements to be an insurance company. Combined, these four companies have a direct written premium of less than one million dollars, which is insignificant compared to the revenue produced from inspections.
The five authorized inspection agencies at the top of the chart are not insurance companies at all, but are third-party inspection companies based on the ASME rule changes implemented in 2010. A significant portion of all new construction inspections are performed by the top two groups, which are in practice or in fact third-party inspection companies and are not insurance companies.
This profile of AIAs represents a huge change from the common practices that have been in place since the 1920s. It's not only that the AIAs have changed; there has also been a significant change in the experience of inspectors and supervisors that perform the actual work. The data you see is HSB Global Standards' data. And while my company's data may not apply to others, at least it gives you one perspective.
Today, 58 percent of our inspectors have less than 10 years of experience as AIs. In the US, most of them are recruited from the nuclear Navy. Outside the US, most of them are degreed engineers. These individuals usually come to us with enough credits to pass the National Board commission, but with very little practical experience with the code or the construction of boilers or pressure equipment.
While I don't have as precise data prior to 2000, most people in this business had very long careers. And based on a 30-year career, one would expect that 33 percent of the people would have less than 10 years of experience. Also in years past, we could hire Navy boiler techs or experienced people from industry and put them in an insurance environment where they live the code. With this significant increase in new and inexperienced people, we have had to provide a significant amount of additional training in order to assure our people are capable of performing high-quality inspections. To gain additional confidence they were performing quality inspections, our training program requirements significantly exceed both ASME and the National Board's requirements.
Along with the AIAs changing and less experienced inspectors, there is also a significant difference in the motivations of an insurance company versus an inspection company. As I said, insurance companies are focused on reducing the risk of loss. For this reason, insurance companies have full-time employees doing full-time ASME work and demanding sufficient inspection be provided in order to manage the risks. Until recently, it was also primarily a North American business.
Inspection companies are focused on making a profit from their inspections just as any other normal provider of goods and services. Controlling costs is a significant part of managing this type of business, so there is much focus on productivity. Inspection companies use inspectors who can inspect many codes, where ASME may only be part of the inspection work they do. They also use contractors as important tools to maintain productivity, and the use of contractors has increased significantly over the last 10 years.
In addition, in a very competitive global market, competing on the amount of inspection time is the easiest way to reduce price and gain market share. The globalization of our customer base has also meant a much higher percentage of our AIs are located outside of the US and in places with much different cultures and expectations, working for customers who have much less experience with the code. This substantially complicates providing a consistent level of quality in our global services.
Everything I have shared with you so far are things I learned since I came to HSB Global Standards. One of the things that I brought with me when I joined the company was a strong Six Sigma background, especially as applied to services. Six Sigma is a set of tools and a strategy for process improvement, and a high-level process is defined by the acronym DMAIC (define, measure, analyze, improve, control).
Briefly, one first defines the critical elements that impact quality, then puts in place measures to determine the performance of the critical quality elements. At this point, the data can be analyzed to determine performance objectives and identify variations. Improvements can then be made, and controls put in place to assure consistency in uniformity.
When I tried to apply Six Sigma concepts at my company, this is what happened. I first asked for the critical quality elements for code inspections. I was referred to our quality assurance manual and the various applicable codes, where there is actually a lot of good information and definitions about the requirements for an authorized inspector. However, when I got to my next question, what are the measures of inspection quality, I found that for the most part there were no quantitative measures available.
Of course, the “analyze,” “improve,” and “control” parts of the DMAIC process were irrelevant since without data you really can't control something. In our very mature business, it took four years to get quantitative measurements in place. Most of that time was related to overcoming internal inertia – Bob Wielgoszinski would be an example – and obstacles within the system.
The metrics aren't defined, nor are they required by the National Board or ASME. Measures of inspection quality had not been an area of focus in the past. But we needed to get started. So I asked our technical leadership to determine the metrics we should use to measure the quality of service delivery. They came up with three to start: monitoring of the customer QA system, daily inspection record, and QAI-1 audits. It should be no surprise that all three of these activities are required by authorized inspection agencies, although they are not required to be measured. I would also point out that only completion of QAI-1 audits are commonly reviewed during an ASME accreditation audit.
[Slide “Monitoring Requirements”] Let me quickly review the requirements. Monitoring is a common requirement throughout the code. It's described in many ways, but they all generally say that the inspector must monitor the certificate holder's quality program. It's in many places, but these are some of the applicable codes and standards requiring monitoring.
[Slide “Bound Diary”] The inspector must maintain a bound diary recording the inspector's inspection activities. This is both a QAI-1 requirement and an NB-263 requirement.
[Slide “NB and QAI-1 Audits”] The one part of the code that does have specific criteria relates to inspector audits. Both the requirement that they be done and the frequency are set forth in NB-263 and QAI-1. Given that there is an actual specified measurement, it should not be surprising that this is one of the most commonly audited items in the triennial accreditation audit. And listed on the slide are some of the relevant code paragraphs for audits.
[Slide “Just to be Clear”] We decided to measure these activities as follows. I would like to make a disclaimer before I show you this information: ASME and the National Board require monitoring, they require bound diaries, they require inspector audits, but the measures are not defined, nor are they required.
[Slide “How We Measured”] The following definitions and measures are HSB Global Standards' measures of itself. For the audit requirements, we measure the number of required audits completed on time. For the bound diary requirements, we review one month of bound diary entries for every one of our inspectors. That's over 10,000 entries per year that we review. The entries were categorized into satisfactory; recommendations on entries that could be improved; unsatisfactory; and then sufficiently unsatisfactory entries (we had to raise an internal NCR against those entries).
Monitoring had the least code definition and required the most decisions on what and how to measure it. We settled on measuring the number of fully active shops where 100 percent of monitoring was performed in a calendar year. Now, we defined a fully active shop as a shop that required at least 16 visits over a three-year period.
[Slide “Results Against Key Performance Indicators”] Here are the results of our measurements. We started doing shop inspections in 1871, and from that time through 2000, our quality metrics were essentially subjective. When I arrived in 2002, the subjective measurement, both internally and externally, was that we were really good. Our actual measurements in 2008 painted a somewhat different picture. Seeing our actual performance was a shock to everyone and definitely not what any of us would consider really good. With only 70 percent of our customers fully monitored, and 75 percent of our required audits completed on time, and over 200 internal NCRs raised on our bound diaries, a call to action was necessary.
Our results for 2012 show improvements in all areas, with monitoring up to 88 percent, our on-time audits up to 90 percent, and far fewer NCRs for bound diary entries. I would like to state once again, this is our performance against our own internal metrics as previously described. Despite this performance against our internal metrics, we are currently (and have been) in full compliance with our accreditation requirements with both the National Board and with ASME.
[Slide “What Does this mean to HSB Global Standards?”] The quality of our service delivery was nowhere near as good as we thought when our evaluation was entirely subjective. Until we measured our performance, there was no strong drive to improve our performance. And the quantitative measures will allow us to demonstrate our performance to accreditors and to our customers. Most importantly, we now understand our own gaps and are able to proactively address those gaps.
[Slide “General Observations”] Having gone through this learning experience and seeing the actual results, I would like to share a few observations with you. Societal desire for transparency is increasing. We see this in new financial regulations like Sarbanes-Oxley, we see this in food nutrition, we see this in workplace safety, and I'm sure you can think of many other examples. Consistent, appropriate measures increase transparency and are completely in keeping with the National Board's mission from 1919. We as inspectors are in the business of validating the quality of products our manufacturers produce, and we should also evaluate our own quality. These metrics allow us to be proactive in improving our quality and maintaining it at an acceptable level while reducing our risks.
I would like to acknowledge the significant actions that the National Board is taking to address some of the challenges I have outlined. Specifically, the new biennial proficiency training requirements for inspectors – this goes a long way to help assure that inspectors maintain a minimum level of expertise throughout their career. The requirement to validate proficiency by testing is an excellent example of having measurements in place. Also, the new requirements for validation of experience for obtaining a commission will go a long way towards having measurable data that provides some confidence that new inspectors coming into the system have the experience required by the National Board. I strongly commend the National Board on these actions in moving this direction.
In conclusion, I have two final thoughts for you to ponder. First, when you look at the next piece of pressure equipment that shows up in your jurisdiction, are you confident that the inspections have been done in a uniform way? How do you know?
Thank you very much.
Question & Answer
Mr. Cantrell: Are you trying to push for a QAI and NB-263 change to maybe address some of these qualitative criteria?
Mr. Bull: The question is if I am pushing for a change in either QAI or NB-263. I think what I want to do is make you aware of the situation. We are in a much different place than we were for most of the time that this process has existed. I think it would be helpful to the industry to have measurements in place, and I think somehow that should be addressed through the conformity assessment documents. Whether what I demonstrated we are doing is the right thing or not, I don't know. I definitely think it is something that should be considered, especially since there are many more people doing inspections and companies that aren't just insurers that have the direct link between risk of loss and the actions that they are taking.
Dr. Lau: I think you would be happy to know that this morning in a meeting in QAI-1, we were talking about feedback on the AI and AIA where the manufacturers are going through the surveys. So that's another small step. But it's good to know that your company is taking the big step in making sure that quality is there, and I would really encourage all the AIAs irrespective, whether they are jurisdictional or insurance or inspection companies, to do likewise.
Mr. Bull: Thank you, Ken. One last note: none of this should be perceived as being negative to the expansion of the AIAs. AIA expansion was important and needed to happen, and we have always supported it. I think the biggest point is that in the past there were many unstated, underlying assumptions that things would work one way. Now, with globalization, things are changing, and issues need to be addressed in a different way. Those previous assumptions now need to be stated and they need to be addressed so that we can take this wonderful quality system we have and assure that it maintains its reputation. Because in the end, it's no better than the conformity assessment side – the code is great, but if it's not enforced, it won't matter.