Truckers say report urging FMCSA to tweak data used for CSA validates motor carriers’ complaints

The long-awaited government scientific report on the data used to compile motor carrier safety ratings for hundreds of thousands of trucking companies is backing truckers complaints that some of that data is junk.

While the Federal Motor Carrier Safety Administration’s (FMCSA) Safety Measurement System (SMS) used to identify commercial motor vehicle carriers at high risk for future crashes is conceptually sound, the report concluded, several features of its implementation need improvement.

That’s according to a new congressionally mandated report from the National Academies of Sciences, Engineering, and Medicine.  The NAS recommended that over the next two years, FMCSA develop a more statistically principled approach for the task, based on an item response theory (IRT) model. That’s an approach that has been applied successfully in informing policy decisions in other areas such as hospital rankings. 

If the model is then demonstrated to perform well in identifying motor carriers that need interventions, the NAS report concluded, FMCSA should use it to replace SMS.

SMS is structured in “a reasonable way” and its method of identifying motor carriers for alert status is “defensible,” the report concluded. But it says much of what is now done is “ad hoc” and based on subject-matter expertise that has not been sufficiently empirically validated.

The NAS recommends FMCSA adopt a more “statistically principled approach” that can include the expert opinion that is implicit in SMS in a natural way.

This was music to the ears of the trucking lobby, which has been saying for years some of the data used in the seven-year-old Compliance, Safety, Accountability program was flawed.

Specifically, American Trucking Associations officials hailed the report, saying it substantiated many of the concerns ATA has raised about the program.

“We appreciate the work the National Academies of Science has done in helping motor carriers, FMCSA and the general public learn about the limitations of CSA,” ATA President and CEO Chris Spear said in a statement. “This report has confirmed much of what we have said about the program for some time: the program, while a valuable enforcement tool, has significant shortcomings that must be addressed and we look forward to working with FMCSA to strengthen the program.” 

Specifically, ATA noted the NAS study validated the trucking industry’s concerns about the inclusion of certain types of violations in the CSA system, that geographic enforcement disparities can have a significant impact on carriers’ scores and that the collection and use of clean inspections is critical to the accuracy of the program.

“We strongly agree with the National Academies’ conclusion that CSA should be based less on the ‘subject-matter expertise’ of enforcement and more on the empirically validated data,” said ATA Director of Safety Policy Sean Garney. “We also see great potential in the Academies’ recommendation that FMCSA overhaul the current CSA methodology in favor of a new, more adaptive, data-centric model with the potential to address serious flaws in the system.

The Compliance, Safety, Accountability (CSA) data-driven safety compliance and enforcement program began in 2010. It was intended to reduce the frequency of commercial motor vehicle crashes, injuries, and fatalities. CSA consists of the Safety Measurement System; resulting interventions; and a Safety Fitness Determination (SFD) rating system, which is used to determine motor carrier safety.

There are about 100,000 fatality- or injury-causing crashes involving large trucks and buses in the United States each year, the report says.  Approximately 3.5 million truck roadside inspections are conducted yearly by specially trained inspectors on approximately 900 potential violations of safety regulations that fall under six categories called “BASICs.” They are unsafe driving, hours of service compliance, vehicle maintenance, controlled substances/alcohol use, hazardous materials compliance, and driver fitness.

FMCSA uses information that is collected mainly during these inspections to identify motor carriers that are operating unsafely and therefore are likely to be at higher risk for future crashes.  Carriers found to have frequent violations are subject to interventions from FMCSA, which can include warning letters and investigations, the report says.

But truckers have complained that the methodology for some of the BASICs is flawed, and the NAS appeared to back them up on some of their complaints.

The report identified a number of data quality issues and recommended that FMCSA should continue to collaborate with states and other agencies to improve the collection of data on vehicle miles traveled and on crashes, data which are often missing and of unsatisfactory quality.  Including vehicle miles traveled data by state and month will enable SMS to account for varied environments where carriers travel -- for example, in icy winter weather in the North.