Patient Deduplication

Although the online interface forces the user to perform a thorough search for a patient, duplications are still possible due to the following reasons that occur while performing a search:

Overall, any difference between how the patient was originally entered and how it is searched for can result in the system not locating an exact match and, therefore, a duplicate patient can be entered.

The search process requires a minimum amount of search criteria to be entered. This information is fed into a sophisticated search algorithm that uses many different combinations of the information against the related database tables. If a patient is not found, the user is allowed to enter the patient as "new."

Batch loads to the registry are sent by organizations that independently maintain their own databases, which are basically their own listings of patients. Since they are maintaining their own databases of records, it is very possible to send duplicates. These duplications must be handled centrally at the registry as they are processed into the system.

When the records are sent via batch load, the data is stored in a holding table called the Pre-Reserve table. The data stays in this table until the Automatic Duplicate Identification Procedure, also referred to as the deduplication process, runs. The deduplication process moves the record, one record at a time, based on the process results. The record's end destinations, however, are the Reserve and Master tables. However, the duplicate identification result determines the record's actual end destination.

Patient Deduplication Processes

Obviously, the goal is to avoid creating duplicate records for the same patient. This process is referred to deduplication or duplicate identification.

There are four processes that can determine whether a record in the database is a duplicate or not. The processes are:

The Manual Duplicate Identification Procedure uses the Manual Deduplication option. This process involves comparing both the incoming record and the database record and making a decision. Thus, this is a manual process.

The Automatic Duplicate Identification Process offers two different options: deterministic or probabilistic. However, the identification process is the same.

The Automatic Duplicate Identification Procedure uses the rule-based algorithm. It looks at the incoming data (IRMS# + Patient ID = Patient Identifier) record in the Pre-Reserve table and attempts to locate a match or similar record.

The image below shows the process, when the process may run, and the database tables a record might encounter.

Database Tables for Record Storage

Table	Description
Pre-Reserve	Holds patient records waiting to be processed into the Reserve and Master tables. When the data is loaded into the Pre-Reserve table, some of the fields are pre-processed. Both the Before and After process records are stored in the table. Addresses are pre-processed to ensure conformity to postal standards and spaces are removed. Names are pre-processed in three ways: Unnecessary spaces and special characters (such as hyphens) are removed Names are checked to see if FIRST (name) and MIDDLE (name) have been swapped, or if MIDDLE and LAST have been swapped. Names are checked to ensure FIRST and MIDDLE names were both entered into the FIRST name field.
Reserve	Stores the patient record exactly as it was sent from the Facility or IRMS. The Facility/IRMS database's IRMS number and patient ID are stored in this table so that the record can be mapped back to the Facility/IRMS database.
Master	Stores the latest patient record sent from a Facility/IRMS database. Each Reserve record that has been mapped to this Master record can reference it. The mapping of patient Reserve records to patient Master records is used to create a composite vaccination record by combining all of the vaccinations sent from the Facility/IRMS databases.
Possible Duplicate	Holds incoming patient records that have possible duplicate matches in the Reserve table. Records only go into this table if the Automatic Duplicate Identification Procedure could not definitely identify a duplicate record. Records in this table require human review using the Manual Duplication Identification or Manual Deduplication process.
Ambiguous ID	Holds incoming patient records that have Patient Identifier problems. Example: an IRMS sent patient 5 twice - referred to as patient 5a and patient 5b. Patient 5a has a different name than patient 5b. Records in this table require human review using the Ambiguous ID application.

Records are moved from the Pre-Reserve table to the Master table once record at a time. The journey of a record to the Master table depends on whether another patient record is found to be similar or a definite match.