Duplicate

Auto-generated

This page is imported automatically from an external repository.

Source project: Open repository

This module provides functionalities for identifying duplicate entries and merging them.

Overview

The duplicate module consists of three layers:

Detection – Scans entities for duplicates using configurable search patterns, backed by Solr (Index-RC), a DB-RC, or a custom jDito-RC.
Display – Shows detected duplicates in the client and allows users to ignore or merge them.
Merging – Merges two duplicate records into one; entity-specific merge providers are configured in DuplicateMerge_lib and can be extended via duplicateMergeProvider_service.

Components

`DuplicateScanner_lib`

The central library for all duplicate detection logic. It contains two classes: DuplicateScannerUtils and DuplicateUtils.

DuplicateScannerUtils provides static helper functions used across the module: reading scan configurations from the DUPLICATESCANNER table, building the data objects passed to the duplicate check, managing the HASDUPLICATE table, and handling ignored duplicates in UNRELATEDDUPLICATES. The functions updateHasDuplicateEntry and updateHasDuplicateEntryForExtern are the main entry points for keeping HASDUPLICATE in sync when records are saved.

DuplicateUtils is instantiated per duplicate check. It reads the scan pattern for the entity, prepares the filter by applying configured modifications (e.g. exclude terms), and determines how the query is delivered based on the RecordContainer type used (see RecordContainer type behavior). execute() returns the list of found duplicates; checkForDuplicates() returns a boolean and is used in contexts where only presence matters.

`IndexsearchFilterUtils_lib` (from `@aditosoftware/utility`)

When the entity uses an Index-RecordContainer, DuplicateUtils does not pass a filter object to the RecordContainer. Instead, it uses IndexsearchFilterUtils to convert the ADITO filter object — together with the fuzzy configuration extracted from the scan pattern — into a valid Solr query string. This is necessary because fuzzy values cannot be cleanly forwarded to Solr via a filter object.

The resulting query string is passed via the entity parameter DuplicateFilterPattern_param to the RecordContainer, where it is appended to the existing index search pattern in the patternExtensionProcess. For DB-RC and jDito-RC, the filter object is passed directly.

Note: The behavior of individual filter fields in the index depends on the configured IndexFieldType. IndexFieldTypes can apply additional transformations (e.g. phonetic matching) that influence duplicate results beyond what the fuzzy value alone suggests. For details see Indexsearch.

`DuplicateMerge_lib`

Provides the merge provider configuration used by the merge dialog. It maps entity names to their merge provider names and defines which fields are excluded from merging globally and per entity. Built-in providers for Person_entity and LetterRecipient_entity are defined here directly; additional providers from other modules are registered via duplicateMergeProvider_service.

`duplicateMergeProvider_service`

Extension point for other modules to register additional merge providers. DuplicateMergeUtils.mergeProvider() collects all registered implementations and merges them with the built-in defaults.

`rebuildDuplicates_serverProcess`

Recalculates all duplicates for a given entity type. It clears all existing HASDUPLICATE entries for that type first, then iterates over all records in batches and recalculates their duplicate status. The batch size is configured via custom.duplicates.dataBlockSize (default: 5000).

The process runs server-side only and cannot be started from the ADITO Manager since it requires a configuration to be passed. It is triggered from the Duplicate Configuration context via the "Rebuild selected entries" action; afterwards "View duplicates" shows the results.

`getDuplicateConditionalListSql`

A static helper on DuplicateScannerUtils that returns a SqlBuilder subselect targeting HASDUPLICATE. It is used to add a "Duplicates" filter extension to an entity's DB RecordContainer so users can filter the list view by duplicate status.

The filter extension is contributed via a modification inserting a filterExtension into the entity's FilterExtensionExtensionPoint:

import { DuplicateScannerUtils } from "DuplicateScanner_lib";
import { newWhere, SqlBuilder } from "SqlBuilder_lib";
import { SqlUtils } from "SqlUtils_lib";

var sqlOperator = SqlUtils.getSqlConditionalOperator(vars.get("$local.operator"));

result.string(newWhere(
    "CONTACT.CONTACTID",
    DuplicateScannerUtils.getDuplicateConditionalListSql(
        ["MyNew_entity"],
        vars.get("$local.rawvalue"),
        sqlOperator, true
    ),
    SqlBuilder.IN()
));

This relies on HASDUPLICATE being up to date — it reflects cached duplicate state, not a live recalculation.

`DuplicateScanner_entity`

Manages the scan configurations stored in the DUPLICATESCANNER table. Shown in the client under the Administration menu group, it allows admins to adjust search patterns and trigger a rebuild.

Adding a new entity to the duplicate functionality is not possible via the client here — it must be done at the code level (see Adding duplicate functionality to a new entity).

Database tables

`HASDUPLICATE`

Caches which records currently have duplicates and how many.

Column	Type	Description
`HASDUPLICATEID`	char(36)	Primary key
`OBJECT_TYPE`	varchar	Entity name, e.g. `Organisation_entity`
`OBJECT_ROWID`	varchar	ID of the record
`DUPLICATECOUNT`	integer	Number of detected duplicates

`DUPLICATESCANNER`

Stores one scan configuration per entity.

Column	Type	Description
`ID`	char(36)	Primary key
`ENTITY_TO_SCAN_NAME`	varchar(200)	Entity to scan, e.g. `Organisation_entity`
`FILTER_NAME`	varchar(200)	Display name in the Duplicate Configuration context
`ID_FIELD_NAME`	varchar(100)	Primary key column of the entity, e.g. `CONTACTID`
`SCAN_PATTERN`	clob	JSON filter object representing the search pattern
`EXTERNAL_SERVICE_USAGE_ALLOWED`	tinyint	Unused — set to `0`

`UNRELATEDDUPLICATES`

Tracks manually ignored duplicate pairs.

Column	Type	Description
`ID`	char(36)	Primary key
`DUPLICATETYPE`	varchar	Entity name
`SOURCEDUPLICATEID`	varchar	ID of the source record
`UNRELATEDDUPLICATEID`	varchar	ID of the ignored duplicate

Scan pattern configuration

The scan pattern is a standard ADITO filter object stored as JSON in DUPLICATESCANNER.SCAN_PATTERN. Admins can edit it in the client via Administration → Duplicate Configuration.

For filter rows using the operators CONTAINS or EQUAL, the key value can be a JSON object to pass additional configuration:

{"fuzzy": 1, "exclude": ["gmbh", "co", "kg", "ag", "bank"]}

fuzzy – Enables fuzzy matching. Only evaluated by the Index-RC implementation; for DB-RC and jDito-RC the value has no effect by default. The integer controls Solr's edit-distance tolerance. The field value is split into individual words by whitespace — each word gets the fuzzy suffix applied independently, and the resulting terms are combined with OR. This means a record matches if any word from the search value matches, not necessarily all of them. Words connected by a hyphen count as a single word. Internally forces the operator to EQUAL. Results are also influenced by the IndexFieldType of the field (see Indexsearch). Fuzzy can be applied to any field that uses EQUAL on free-text content — the default scan pattern for organisation names already demonstrates this.
exclude – Array of strings stripped from the field value before comparison (regex-based). Primarily used to remove legal-form suffixes like "GmbH" or "AG" from the search value before the query runs. This serves two purposes depending on the operator: with fuzzy (OR across words), a shared suffix like "GmbH" would cause "Consulting GmbH" and "Implementation GmbH" to match — a false positive. Without fuzzy, "Bauer GmbH" and "Bauer AG" would not match at all because the full string differs — a false negative. Stripping the suffix avoids both.

Only the operators CONTAINS, CONTAINSNOT, EQUAL, NOT_EQUAL, and ISNULL are supported. Unsupported operators are silently replaced by EQUAL. To add support for a new operator, the following places must be updated:

DuplicateScannerUtils.getPermittedFilterOperator() — add the operator to the allowed list, otherwise it gets replaced before any further processing
IndexsearchFilterUtils_lib (IndexsearchFilterRow.fromFilter) — define the corresponding Solr behavior (Index-RC only)
DuplicateUtils.prototype._modifyFilterForProviderType — add RC-type-specific handling if needed

RecordContainer type behavior

RC type	How filter is delivered	Fuzzy handling	Unsupported operator fallback
Index	Solr query string via `DuplicateFilterPattern_param`	Extracted into fuzzy object; `IndexsearchFilterUtils` appends `~n` per word	Replaced with `EQUAL`
DB	Standard ADITO filter object	Not applied — value is ignored	Replaced with `EQUAL`
jDito	Standard ADITO filter object	Value written to filter object; must be handled by the custom implementation	No automatic fallback
Dataless	—	—	Throws error

Adding duplicate functionality to a new entity

1. Register in `DUPLICATESCANNER`

Add a Liquibase changeset inserting one row into DUPLICATESCANNER. The scan pattern can initially be left empty and configured later by an admin in the client.

<insert tableName="DUPLICATESCANNER">
    <column name="ID" value="<new-uuid>"/>
    <column name="ENTITY_TO_SCAN_NAME" value="MyNew_entity"/>
    <column name="FILTER_NAME" value="MyNewDuplicates"/>
    <column name="ID_FIELD_NAME" value="MYNEWID"/>
    <column name="EXTERNAL_SERVICE_USAGE_ALLOWED" valueNumeric="0"/>
    <column name="SCAN_PATTERN"
        value='{"filter": {"type": "group", "operator": "AND", "childs": []}}'/>
</insert>

2. Create a RecordContainer with the provider `"Duplicates"`

In MyNew_entity, create a RecordContainer (Index-RC, DB-RC, or jDito-RC) and set its provider name to exactly "Duplicates". DuplicateUtils looks up this provider by name at runtime and determines the RC type from it.

3. Hook into `onDBInsert` / `onDBUpdate`

Call DuplicateScannerUtils.updateHasDuplicateEntry to keep HASDUPLICATE in sync whenever a record is saved:

import { DuplicateScannerUtils } from "DuplicateScanner_lib";
import { EntityUtils } from "EntityUtils_lib";

DuplicateScannerUtils.updateHasDuplicateEntry(EntityUtils.getCurrentEntitytId());

4. Connect the duplicate display (main view)

Add a consumer to the entity referencing Duplicate_entity with field name Duplicates and selection mode MULTI. In the valueProcess of the parameter DuplicateObject_param, pass the current record's field values using DuplicateScannerUtils.getDataForDuplicateCheck — all fields referenced in the scan pattern plus the primary key must be included:

import { DuplicateScannerUtils } from "DuplicateScanner_lib";
import { EntityUtils } from "EntityUtils_lib";

result.object(DuplicateScannerUtils.getDataForDuplicateCheck(
    EntityUtils.getCurrentEntitytId(), {
        MYNEWID: vars.get("$field.MYNEWID"),
        MYFIELD1: vars.get("$field.MYFIELD1"),
        // all fields referenced in the scan pattern + primary key
    }
));

All fields that are (or could be) referenced in SCAN_PATTERN must be included here, along with the primary key. The rebuild path reads these fields automatically via entities.getRows(); this valueProcess is the equivalent for the UI context.

Use DuplicateFilter_view in the main view via this consumer.

5. Connect the duplicate check in the edit view

Use HeaderFooterLayout and add DuplicateEdit_view via the same consumer.

6. Index-RC only: wire up `DuplicateFilterPattern_param`

If the entity uses an Index-RecordContainer, the duplicate check passes the Solr query string via the entity parameter DuplicateFilterPattern_param instead of a filter object. The parameter must be declared on the entity with expose = true.

To integrate it into the index search, declare a service on the entity (e.g. myNewIndexPatternExtension_service) and call it at the end of the patternExtensionProcess, following the same pattern as Organisation_entity:

var pattern = indexsearch.buildPattern(patternConfig);

modules.loadService("myNewIndexPatternExtension_service")
    .forEach((service) =>
    {
        let extendedPattern = service()(pattern);
        if (Utils.isNotNullOrEmptyString(extendedPattern))
            pattern = extendedPattern;
    });

result.string(pattern);

The service implementation in this module reads the parameter and appends it to the existing pattern with AND:

import { vars } from "@aditosoftware/jdito-types";
import { Utils } from "@aditosoftware/utility/process/Utils_lib/process";

const idxPattern = vars.exists("$param.DuplicateFilterPattern_param")
    ? vars.get("$param.DuplicateFilterPattern_param") : "";

let res = (pPattern) =>
{
    if (idxPattern)
    {
        if (Utils.isNotNullOrEmptyString(pPattern))
            pPattern += " AND ";
        return pPattern += idxPattern;
    }
};

//@ts-ignore
return res;

Note: The index fields returned by query.js of the entity are used by the duplicate scanner. Adding new fields is safe, but removing or renaming existing ones may break the configured scan pattern. See the comment at the top of Organisation_entity/recordcontainers/index/query.js for reference.

7. Add a "Duplicates" filter to the list view

To let users filter the entity list by duplicate status, add a filterExtension to the entity's DB RecordContainer via a modification. The filterConditionProcess uses getDuplicateConditionalListSql to query HASDUPLICATE:

import { DuplicateScannerUtils } from "DuplicateScanner_lib";
import { newWhere, SqlBuilder } from "SqlBuilder_lib";
import { SqlUtils } from "SqlUtils_lib";

var sqlOperator = SqlUtils.getSqlConditionalOperator(vars.get("$local.operator"));

result.string(newWhere(
    "MYTABLE.MYNEWID",
    DuplicateScannerUtils.getDuplicateConditionalListSql(
        ["MyNew_entity"],
        vars.get("$local.rawvalue"),
        sqlOperator, true
    ),
    SqlBuilder.IN()
));

This filter reflects the cached duplicate state in HASDUPLICATE. It does not trigger a live recalculation.

Alternative: trigger a duplicate check directly in a process

If the duplicate check is needed outside of a UI context — for example in an import script or server process — DuplicateUtils can be instantiated directly without any entity consumer setup:

import { DuplicateScannerUtils, DuplicateUtils } from "DuplicateScanner_lib";

var mappingObjects = DuplicateScannerUtils.getDataForDuplicateCheck(
    "MyNew_entity", {
        MYNEWID: "some-id",
        MYFIELD1: "some-value"
    }
);
var duplicates = (new DuplicateUtils(mappingObjects[0])).execute();

This is also how rebuildDuplicates_serverProcess works internally.

Overview​

Components​

DuplicateScanner_lib​

IndexsearchFilterUtils_lib (from @aditosoftware/utility)​

DuplicateMerge_lib​

duplicateMergeProvider_service​

rebuildDuplicates_serverProcess​

getDuplicateConditionalListSql​

DuplicateScanner_entity​

Database tables​

HASDUPLICATE​

DUPLICATESCANNER​

UNRELATEDDUPLICATES​

Scan pattern configuration​

RecordContainer type behavior​

Adding duplicate functionality to a new entity​

1. Register in DUPLICATESCANNER​

2. Create a RecordContainer with the provider "Duplicates"​

3. Hook into onDBInsert / onDBUpdate​

4. Connect the duplicate display (main view)​

5. Connect the duplicate check in the edit view​

6. Index-RC only: wire up DuplicateFilterPattern_param​

7. Add a "Duplicates" filter to the list view​

Alternative: trigger a duplicate check directly in a process​