Bad Data Exists. What Can AI Do About It?
Dr. Bob Lindner is the Chief Science and Technology Officer at Veda, a company addressing provider directory data challenges.
It’s no surprise to anyone who works with data—it’s messy. In every industry and every business, there are data anomalies and issues that can impact the story data tells. If we have any hope of improving data practices and making collected data truly actionable, we first have to acknowledge its limitations and then explore modern solutions for improving it.
Bad Data Is The Norm
With the new federal administration exploring cost-cutting measures and releasing data nearly daily, a specific example caught my eye—it was a Social Security disbursements by age graph, with the data suggesting 210 year olds are receiving Social Security entitlements. As a data scientist who has been working with healthcare data for over 10 years, this graph wasn’t shocking to me.
I recently saw one dermatologist who was practicing at 20 different variations of one address; imagine the extra legwork required by a patient to find out where you are booking an appointment. Or how about two providers with the exact same name but one is a veterinarian on the West Coast and the other is a physician in New York? There is state licensing info for both of them, but the only one with a federal National Provider Identifier (NPI) is the veterinarian. These are complex data problems occurring every day.
Data engineers know that a lot of data in every industry is collected manually, and this often introduces errors that are quickly propagated and magnified throughout downstream processes. In fact, most data systems in the modern economy, all around the globe, have shockingly out-of-date practices. With a spotlight on data issues right now, it’s important to dig deeper and examine data processes to have any hope of modernizing databases and making data functional.
Modernizing Data: Where To Start?
In my world of provider directories—which include providers’ names, addresses, phone numbers and more—typical methods for updating data are antiquated and include things such as call campaigns, attestation from providers or their health groups and manual roster updates. The methods are labor-intensive, slow and have a high likelihood of errors. The industry needed to evolve beyond asking physicians and health plans to manually report in and validate information.
My advice to anyone seeking to clean up data in any industry? First, modernize the systems that create, store and use the data and address the methods used for data collection. Fixing bad data problems is just like dealing with any mess: it’s essential to cleanse and filter. Think of a house filled with clutter. First, you need to clean out the house and keep only the essential items. Then you can do a deep clean. But to keep it looking great, you have to go one step further. You need to be diligent that any new items that enter the house are also useful, or you’ll deal with this problem again. To realize the full potential of modernized methods, start in the cleanest position possible.
Here’s Where AI Comes In
As often is the case, technology can be relied on as a new way to solve old problems. Automation and AI can do what humans can’t do or wouldn’t be able to do efficiently. I’m fond of saying, “If manual solutions could successfully process provider data, it would have worked by now.” In my industry, AI has been embraced to a point, but there are regulatory barriers that rely on old methods of data collection. Changing these regulations to include less error-prone methods of data collection—one that does not require physicians and health plans to manually report and validate data is a good place to start for advancing data practices. Truly, only AI can solve bad data problems.
The Right Data Avoids Blindspots
The right data solves problems while avoiding blind spots. For example, when answering the question, “Why is it difficult to see a healthcare provider in my area?” it’s easy to assume it’s because there are not enough providers in an area, and a cursory search might prove that. But that’s not the whole story.
With high-quality data, you can determine if that’s truly the case or if it’s difficult to get an appointment because the available providers are not displayed and categorized correctly. Or even, if there are enough providers accepting patients but they are scheduling so far out that they might as well be unavailable. Without high-quality data and a modern system to examine data, the answer to the question, “Why is it difficult to see a provider in my area?” is just a guess. Depending on what the reliable data tells you, the path forward may be recruiting more providers or it may be accurately displaying the providers that are already practicing.
The Importance Of Validation
After modernizing tech and answering the questions you seek the answers to, inspect your work. In a data-driven landscape, validation isn’t just a checkbox; it’s a strategic imperative for success. From mitigating risks to maximizing opportunities, I encourage everyone to be open to third-party verification. When validation is embraced, teams are empowered to deliver feedback and growth opportunities are more apparent. Customers want to work with tested vendors. When they’re making large investments, validation fosters trust.
Not to mention, by validating data claims, there is a safeguard for risks and uncertainties. Open and transparent data practices lead to trust and accountability—something AI desperately needs.
When Bad Data Is Fixed
When messy data is cleaned and accuracy is gained, it’s a game-changer. In healthcare, where the stakes couldn’t be higher, data accuracy plays a key role in achieving quality, affordable healthcare. With provider data accuracy, patients have greater access to care. When it is collected efficiently with modern methods, data is not a burden to doctors nor an obstacle to patients. As seen in these healthcare examples, modern data practices can benefit every industry. Investing energy and time into modernizing data systems and validating outcomes will benefit us all.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?