• NASEM report sees a few flaws, but SMS is structurally sound
  • System ranks carriers based on safety metrics
  • Increased data collection will help with gathering data

A 2017 report by the National Academies of Science, Engineering, and Medicine (NASEM) had the mission of reviewing the Federal Motor Carrier Safety Administration’s (FMCSA) current Compliance, Safety, and Accountability (CSA) program.

In May of 2019, NASEM conducted another study, this time focusing more on driver fatigue and how data collection procedures can be used to determine long-term driver health. Methodologies from the first study were similar for both reports.

The findings of NASEMs report revealed the SMS to be reasonably structured. But the report also suggested that practical validation of some aspects of the SMS are still insufficient and requires improvements.

How the Safety Measurement System (SMS) works

FMCSA’s CSA program aims to ensure that Commercial Motor Vehicle (CMV) carriers keep high safety standards in freight and passenger transportation.

The program uses its safety measurement system as a prioritization tool that ranks motor companies on their safety ratings. The agency then targets low-ranking carriers for intervention programs to reduce the possibility of future crashes.

The rating method of the safety measurement system is based on the frequency and class of violations assessed primarily over roadside inspections from the most recent two-year period.

BASICS and ranking

The violations are divided into six groups that each have their own weight and scoring metrics. The SMS integrates these six measures with another weighted measure, crash frequency. All together, these seven categories are known as the BASICs, which stands for Behavioral Analysis and Safety Improvement Categories.

Carriers who have a multitude of crashes, inspections, and violations have their seven BASICs computed. The resulting calculation ranks them among their peer groups. Larger values mean those carriers have a higher crash frequency or violation count as compared to the rest.

Those carriers who rank above the thresholds set by FMCSA are given a range of interventions, which include warning letters, investigations, fines, and in a worst-case scenario, suspension of operations.

Currently, Commercial Vehicle Safety Alliance (CVSA) trained officers inspect over three million CMVs each year to determine whether any of them are operating in violation of the mandated safety regulations.

Data on CMV crashes, inspections, and violations are logged into the Motor Carrier Management Information System (MCMIS), and then the SMS retrieves its data input from the MCMIS for calculations.

FMCSA’s Safety Measurement System reviews

When the report was released, stakeholders, outside reviewers, and other groups and individuals criticized the safety measurement system for multiple reasons, including:

  • Utilizing assessments that are highly variable
  • Absence of fault consideration in the weighted crash count
  • Grouping carriers in the same peer groups despite different tasks and operations
  • Using measures sensitive to varying effects from different states
  • Employing metrics that do not predict the future crash frequency of carriers
  • Using measures that do not reflect the safety performance improvement efforts of carriers over time

Because of the value at risk, FMCSA recognized the need to examine whether the criticisms were indeed warranted. They also wanted to analyze the performance of the SMS and note any improvements that needed to be made.

How the safety measurement system study came about

The Fixing America’s Surface Transportation (FAST) Act of 2015 had previously recommended the FMCSA fund a study by NASEM to evaluate the safety measurement system.

As a result, two units in NASEM collaborated to form a review panel and begin the study, the Transportation Research Board and the Committee on National Statistics.

The team was tasked to analyze the ability of the SMS metrics to identify the difference between high-risk and low-risk carriers. They were also entrusted to assess SMS’ public usage and then review the data and processes used in calculating measures.

Once the findings were complete, the team provided their advice on data collection and safety measurement methods.

The panel evaluated all of the items below regarding their use and feasibility as part of the current safety measurement system:

  • Many factors that contribute to crashes are not present in the MCMIS
  • MCMIS data as input for creating a crash prediction model that relies on carrier-level behavior is counter-productive
  • FMCSA’s approach with the SMS focuses more on crash prevention than predictions
  • SMS identifies carriers that place little priority on safety performance practices
  • Intervention is aimed at behavior modification that hopes to reduce possible future crash
  • The panel believes that SMS’ general approach is sound
  • In response to the criticisms, researchers found the current SMS implementation as defensible, fair, and not overtly biased
  • The panel believes some implementation details lack the support of empirical studies and are ad hoc
  • The researchers argue that replacing the current algorithm with a statistical model would address implementation details

The panel concluded that the SMS is currently structured reasonably well and can be defended from criticism. But the researchers also acknowledged that there is room for improvement by adopting an approach that is more statistically principled compared to the current SMS.

Similarities between SMS and IRT models

The researchers noted similarities in the approach used by SMS with Item Response Theory (IRT) models.

IRT shows the relationship between a latent trait and the probability of giving a particular response. The latent trait is assumed to directly influence that particular response.

IRT systems have been applied in other contexts and have shown to be both fruitful and efficient.

Because of potential advantages in using IRT, the researchers recommended that a new model be developed. If the system performs well in identifying high-risk carriers for alert, the researchers felt that FMCSA should replace the SMS with it. The new, revised system has not yet been developed or implemented.

According to the panel, the proposed IRT model has the following benefits over the SMS.

  • System scoring uses current observed data instead of dated empirical information
  • Calculation transparency is enhanced
  • Variances of scores and ranks can be directly estimated
  • The likelihood of being chosen for inspection is accounted for
  • A basis on the impact of data insufficiency on carrier safety ratings can be established
  • It justifies evaluating the current structure of the BASICs, including which violations belong to which basic
  • A natural way of subdividing the system into more layers can be naturally explored with IRT models
  • Safety as a multidimensional construct can be investigated, determining the number of BASICs needed from the factors
  • Time can be accounted for, and proper time weights can be integrated
  • New safety measures can be incorporated when available without requiring a rework of the whole system
  • Ranges to better grasp ranking overlaps can be created
  • Changes in safety can be adapted over time

What’s missing and what can be improved?

  • The researchers noted areas in the MCMIS data that could be improved.
  • Because an update on carriers’ Vehicle Miles Travelled (VMT) and Average Number of Power Units (APU), which is the number of CMVs owned or leased by carriers, is only needed every two years, non-recency of figures is possible. Flawed or out-of-date data affects carrier percentile ranks in the SMS.
  • Crash data collection procedures are not standardized across states. A substantial chunk of crash data is also missing from the MCMIS.
  • A good deal of information written in police reports is not depicted in the MCMIS.

The panel also recommended collecting VMT and APU data by state and by month. They also advised the Model Minimum Uniform Crash Criteria (MMUCC) to be universally adopted by the FMCSA to standardize crash data collection.

The panel believes that carrier processes and practices may predict future crash rates. MCMIS, however, lacks information on company operations.

The researchers noted that the SMS believes that carrier operating procedures partly contribute to a significant amount of crashes.

The panel recommends that the following information is obtained.

  • Turnover rate of company employees
  • Type of cargo being handled
  • Level and method of compensation
  • Better data on exposure, such as high-quality VMT from Electronic Logging Devices (ELDs).

Since ELDs are now mandatory for most carriers, definitive VMT data can be conveniently produced.

What’s next?

With the ELD Mandate now in full effect, more information can be obtained for safety measures analysis.

Some ELDs can accurately measure and monitor vehicle performance and anticipate possible service needs. Motive ELDs offer advanced measurement features.

Motive ELDs not only capture vehicle performance but also track the driving style of individual truckers. They monitor events for excessive acceleration, hard braking, and hard cornering. In addition, our ELDs offer the following features and benefits:

  • Audible warnings when it is time to rest
  • Liability reduction or exoneration by providing valuable evidence
  • Reduced insurance costs
  • Expedited inspections
  • The ability to reduce downtime, which means less paperwork and more road time