Skip to main content

XRM Adito Faker

Auto-generated

This page is imported automatically from an external repository.

Source project: Open repository

This module is used to generate huge amounts of fake data for performance and load tests within an ADITO xRM project.

It ships the AditoFaker runner scripts, a project-wide tableDefinition and a template configuration.json, so mass data can be generated through a few npm run scripts. The underlying data-generation engine is the NPM package @aditosoftware/adito-faker, which is pulled in as a devDependency.

Note: AditoFaker is a development tool. Data cannot be generated from the client. Access to the ADITO Designer, the target database and a valid AditoFaker configuration are required.

Contents​

What is Faker?​

Faker is a well-known library for generating fake data (names, addresses, dates, numbers, ...) for testing purposes. It has ports in many languages (PHP, Python, JavaScript, Java, ...).

Within the ADITO context, the Node.js package @aditosoftware/adito-faker combines Faker.js and Sequelize (for database handling) into a tool that can generate massive amounts of realistic test data and write it directly to an ADITO database. This module packages that engine with project-wide xRM configurations so it is ready to run inside an ADITO project.

Prerequisites​

  • ADITO Designer >= 2022.0 (implemented and tested from 2022.0 upwards; earlier versions may work but functionality is not guaranteed).
  • Installed Node.js Designer plugin.
  • ADITO xRM >= 2022.2.0. For older versions, follow the manual setup.
  • Access to the target database: active connection, host, port, user, password.

AditoFaker can also be used outside the Designer (e.g. from VS Code or the CLI) with a manual Node.js installation. In that case, install Node.js locally and run the scripts from your terminal.

Installation​

If the xRM version is >= 2022.2.0, all necessary files are already present. Run npm install from the Designer (right-click → Node.JS → npm install) or from the shell to install the required dependency:

"@aditosoftware/adito-faker": "^1.1.0"

After the install finishes, the AditoFaker scripts appear under Scripts in the Designer's execute window.

Project structure​

All AditoFaker configuration lives in tools/AditoFaker/:

  • tools/AditoFaker/configurations/
    • configuration.json — runtime configuration (DB connection, data tree to generate, logging, ...).
    • tableDefinition.json — database schema + per-field default faker configuration.
    • tableDefinition.csv — optional simplified view of tableDefinition.json, easier to edit in bulk.
  • tools/AditoFaker/scripts/
    • AditoFaker.js
    • TableDefinition_generateCsv.js
    • TableDefinition_syncFromAod.js
    • TableDefinition_syncFromCsv.js

Available scripts​

The scripts are wired into package.json and can be executed from the Designer's Scripts panel or via npm run <name>:

ScriptPurpose
AditoFaker-syncFromAodReads the project's Data_alias.aod and syncs tableDefinition.json: new tables/fields are added, deleted tables/fields are removed. Existing faker configuration for a table is preserved.
AditoFaker-generateCsvGenerates tableDefinition.csv from tableDefinition.json for easier bulk editing.
AditoFaker-syncFromCsvWrites the edited tableDefinition.csv back into tableDefinition.json.
AditoFaker-generateDataReads configuration.json, generates and inserts the mass data into the configured database.

Sync the TableDefinition with the Data_alias​

tableDefinition.json contains the database schema AditoFaker works against. Run AditoFaker-syncFromAod to align it with the current Data_alias.aod:

  • Tables and fields no longer present in the alias are removed.
  • New tables and fields are added.
  • Existing faker configuration for known tables is kept.

To point the sync at a non-default Data_alias.aod, use the paths property in configuration.json.

Editing the TableDefinition as CSV​

For bulk edits, it is easier to work with the CSV representation. Run AditoFaker-generateCsv to produce tools/AditoFaker/configurations/tableDefinition.csv.

When editing the CSV, use LibreOffice Calc with ; as the field separator and ' as the text delimiter. A plain-text editor such as Notepad++ works too. Changing these delimiters will break the reverse sync.

Configuring the TableDefinition​

Each field in the tableDefinition can define one of the following properties. If multiple are set, the first one in this order wins:

  1. faker
  2. relation
  3. value

Additionally, ignore: true can be used to permanently skip a field.

Every field entry also carries a DataTypes object (e.g. { "name": "STRING", "precision": 30 }). It is populated automatically by AditoFaker-syncFromAod from the Data_alias.aod and should not be edited manually.

faker property​

The faker property is the most flexible. It gives access to the entire faker API plus the ADITO-specific extensions.

Simple call (no params)​

For a function without parameters, just set the function name (without the leading faker.):

TABLE          | COLUMN | faker
organisation   | name   | company.companyName

Extended object form​

For anything more complex, use a JSON object:

{
  "name": "modulename.functionname",
  "params": ["param1", "param2", "paramN"],
  "unique": "TABLENAME.FIELDNAME"
}
Sub-propertyRequiredDescription
nameyesFull faker function name, e.g. lorem.sentences, datatype.uuid, adito.keyword.
paramsnoPositional arguments passed to the faker function (in the given order).
uniquenoPrevents duplicate values. Either true (unique within the same field) or "TABLE.FIELD" — the given field is loaded from the DB and used as the initial set of "used" values. Each newly generated value is added. Uses faker.helpers.unique internally and retries up to 10 times before failing.

Example — unique customer code that also respects already-existing rows:

{
  "name": "datatype.number",
  "params": [{ "min": 1000, "max": 999999 }],
  "unique": "ORGANISATION.CUSTOMERCODE"
}

Example — generate 10 lorem sentences:

TABLE          | COLUMN | faker
organisation   | info   | {"name":"lorem.sentences","params":[10]}

relation property​

Reuses a value from a field of the current dataset or of a parent dataset (the parent must have been processed first). Typical use case: sharing a foreign-key UID between the parent record and its children.

TABLE   | COLUMN   | relation
contact | date_new | organisation.date_new

value property​

The simplest option: a fixed literal value that is written to the field for every generated dataset.

TABLE   | COLUMN   | value
contact | user_new | faker

ignore property​

Set "ignore": true on a field to make AditoFaker skip it permanently — no value is generated, no value is inserted. This is useful for computed columns, LOBs like pictures, or fields that must stay NULL. It is stronger than excludedFields in configuration.json, which only skips fields for a single run.

Example (from organisation.salesarea in this module's tableDefinition.json):

{
  "ignore": true,
  "DataTypes": { "name": "SMALLINT", "precision": 5 }
}

Sync the CSV back to JSON​

After editing tableDefinition.csv, run AditoFaker-syncFromCsv to write the changes back into tableDefinition.json. If the CSV delimiters were changed accidentally, the parser will error out — restore the delimiters and re-sync.

Executing AditoFaker​

Once the TableDefinition is configured, adjust the configuration.json to describe what should be generated and where to write it.

Set the database credentials before the first run​

The shipped tools/AditoFaker/configurations/configuration.json contains a placeholder connection string:

mariadb://user:password@localhost:3306/cloud_data_alias

Replace user, password, localhost:3306 and cloud_data_alias with the values of the target database before running AditoFaker-generateData. For an ADITO SSP cloud database, open a tunnel first (Designer or tunnel.bat from the SSP) and keep localhost as the host.

Do not commit real credentials. Keep configuration.json changes local or use a per-environment copy.

Modularized projects: point paths.aod at the transpiled alias​

In a modularized ADITO project the aliases of all modules are merged during the build into dist/aliasDefinition/Data_alias/Data_alias.aod. AditoFaker's default lookup for Data_alias.aod at the project root will miss those merged tables/fields, so paths.aod must be pointed at the transpiled file.

Add paths next to dbConnection in configuration.json:

"paths": {
    "aod": "D:/project/dist/aliasDefinition/Data_alias/Data_alias.aod"
}

Adjust the path to the dist/ folder of the consuming project. After the change, run AditoFaker-syncFromAod to update tableDefinition.json against the merged alias before generating data.

configuration.json properties​

{
  "locale": "de",
  "dbConnection": "mariadb://dbuser:dbpassword@localhost:3306/aditodata",
  "data": {},
  "progressLog": false,
  "sequelizeLog": false,
  "paths": {}
}
PropertyRequiredDescription
localenoFaker locale to switch between languages (e.g. de, en, fr).
dbConnectionyesSequelize connection string. The example above targets MariaDB. Other dialects supported by Sequelize can be used. For an ADITO SSP cloud database, open a tunnel to the system (via the Designer or tunnel.bat from the SSP) and connect AditoFaker to localhost.
datayesTree of table objects that describes which data (and how much) to generate — see below.
progressLogno (default false)If true, prints a log line for every insert-batch call and every finished insert batch (50 000 rows per batch). Recommended for large runs (> 1 000 000 rows) to see whether/when the DB starts slowing down.
sequelizeLogno (default false)If true, prints all Sequelize statements (SELECTs, INSERTs, errors, ...). Very noisy — use only for debugging.
pathsnoOverrides the default file paths, e.g. { "aod": "./path/Data_alias.aod", "tableDefinition": "./path/tableDefinition.json" }. Paths starting with ./ are resolved from the project root (where AditoFaker is executed).

data table objects​

data is a JSON tree. Each key is a table name whose value is a table object describing how to generate rows for that table. Table objects can be nested recursively via relations.

A table object supports the following properties:

{
  "count": 1,
  "excludedFields": ["field1", "field2"],
  "specialFields": {},
  "relations": {}
}
PropertyDescription
countNumber of datasets to generate. A fixed number (10) or a range ([1, 10] — random between 1 and 10). Default 1. In a nested relations block, count applies per parent dataset.
excludedFieldsArray of fields (from the TableDefinition) to skip only for this run.
specialFieldsOverrides the TableDefinition's faker / relation / value for the listed fields only for this table object. Same field syntax as the TableDefinition.
relationsNested table objects. Children have access to all fields of every parent dataset above them (e.g. organisation → contact → address — address can relation any organisation or contact field).

Example — 10 organisations, each with a contact:

{
  "data": {
    "organisation": {
      "count": 10,
      "excludedFields": ["salesarea", "picture", "date_edit", "user_edit"],
      "specialFields": {
        "info": { "value": "generated by AditoFaker" }
      },
      "relations": {
        "contact": {
          "count": [1, 3],
          "excludedFields": ["person_id", "lettersalutation"]
        }
      }
    }
  }
}

Example specialFields object:

{
  "field1": { "value": "value1" },
  "field2": { "relation": "table.field" },
  "field3": { "faker": "fakerModule.functionName" },
  "fieldN": { "faker": { "name": "modulename.functionname", "params": [] } }
}

Multiple sets for the same table (aliases)​

To generate several independent sets for the same table under the same parent, append $<alias> to the key. Every alias is treated as its own table object.

{
  "relations": {
    "address$office":  { "specialFields": { "addr_type": { "value": "OFFICEADDR" } } },
    "address$home":    { "specialFields": { "addr_type": { "value": "HOMEADDR"   } } }
  }
}

Relate to an aliased entry by its aliased name, e.g. "address$home.address".

Nested count values multiply. organisation.count = 100 with contact.count = [1, 5] yields between 100 and 500 contacts — plus every child level below that. Keep an eye on the totals.

Running the generator​

Run AditoFaker-generateData (from the Designer or via npm run AditoFaker-generateData).

The tool starts, tests the DB connection, evaluates the data tree and prints an estimate:

FakerAdito: initialized configuration. Estimated datasets to generate: 299.500:
-> organisation: 1.000
-> contact: 13.500
-> activity: 108.750
-> activitylink: 108.750
-> address: 14.500
-> communication: 40.500
-> person: 12.500
Database connection has been established successfully.
FakerAdito: start generating data.

Once a table reaches 50 000 generated datasets, the batch is handed to an insert queue. Internally: 4 threads × 5 connections × 5 inserts of 10 000 rows each — everything runs asynchronously across Node worker threads.

When the run finishes, a summary is printed with the elapsed time and the true number of rows inserted (may differ from the estimate because of random count ranges):

FakerAdito: finished generating data, waiting for insert to finish.
FakerAdito: finished execution after: 00:01:00.
Inserted 298.118 datasets:
-> activity: 108.125
-> activitylink: 108.125
...

Benchmarks and operational notes​

  • Throughput: during internal tests (local PC, VS Code, standard SSP MariaDB via SSH) the tool generated 50 000 – 300 000 rows per minute. Numbers were measured against a fresh, near-empty database.
  • Large runs (> 7 million rows): insert performance degrades noticeably around this mark on a standard SSP MariaDB — batches of 10 000 rows can take 15 minutes or more. Prefer several smaller runs over one huge one.
  • RAM & relation depth: deep relation trees hold all parent datasets in memory until every child is generated. The first 50 000-row batch for the top table is the slowest. Keep configurations lean and split them if needed.
  • Progress log: enable progressLog for anything above ~1 000 000 rows so you can see when/if the DB slows down.
  • Sequelize log: only use it for debugging — output becomes enormous.

ADITO-specific faker functions​

In addition to the standard faker API, the adito module ships four ADITO-specific functions. They are called exactly like normal faker functions (see the extended object form above).

adito.keyword​

Picks a random active KEYID from an ADITO keyword container.

Signature: faker.adito.keyword(pContainerName: string): string

ParameterTypeDescription
pContainerNamestringAB_KEYWORD_CATEGORY.NAME — the keyword container to pick a random keyid from.

Example:

faker.adito.keyword("OrganisationType"); // -> "ORGPARTNER"

Configuration:

{ "name": "adito.keyword", "params": ["OrganisationType"] }

adito.pickFromDb​

Picks a random non-null value from a database column. The result set is loaded once per key and cached in memory for the rest of the run — so keep the selected set small.

Signature: faker.adito.pickFromDb(pTableColumn: string, [pCondition]: string): string

ParameterTypeDescription
pTableColumnstringTABLE.FIELD from which to load the values.
pConditionstring (optional)Additional SQL condition to restrict the loaded rows.

Examples:

faker.adito.pickFromDb("SALUTATION.SALUTATION", "SALUTATION.ISOLANGUAGE = 'deu'"); // -> "Herr"
faker.adito.pickFromDb("CONTACT.CONTACTID"); // -> "441c7327-6857-4cbd-9159-3f9577651156"

adito.increaseNumber​

Returns a monotonically increasing number for each generated dataset — useful for auto-numbered fields such as customer numbers.

Signature: faker.adito.increaseNumber(pParams?: object): number

Params object:

FieldDescription
keyCache key under which the counter is stored. If omitted, tableColumn (or the current TABLE.FIELD) is used.
startExplicit start value.
tableColumnTABLE.FIELD — if given and no start is set, MAX(column) + 1 is used as the starting value.
conditionOptional SQL condition when computing the initial MAX(...). Placeholders may reference fields of the current dataset.

Example: counter continuing after the current max customercode in the database:

{
  "name": "adito.increaseNumber",
  "params": [{ "tableColumn": "ORGANISATION.CUSTOMERCODE" }]
}

adito.maybe​

Wraps faker.helpers.maybe so it can be used from configuration files. It conditionally executes another faker function (or returns a fixed value), otherwise returns a fallback.

Signature: faker.adito.maybe(pFunctionName: string, pFunctionParams?: any[], pOwnParams?: object, pFallBack?: string|number): string|number|void

ParameterDescription
pFunctionNameName of the faker function to execute in the success case. Use the special value "value" to return pFunctionParams[0] directly.
pFunctionParamsParameters passed to the wrapped function.
pOwnParamsOptions object forwarded to faker.helpers.maybe (e.g. { probability: 0.3 }).
pFallBackFallback returned when maybe decides not to execute.

Example: 30 % chance to generate a mobile number, otherwise null:

{
  "name": "adito.maybe",
  "params": ["phone.number", [], { "probability": 0.3 }]
}

Supported tables​

Base configurations shipped with xRM (since release 2022.2.0):

TableSince
ADDRESS2022.2.0
ACTIVITY2022.2.0
ACTIVITYLINK2022.2.0
COMMUNICATION2022.2.0
CONTACT2022.2.0
ORGANISATION2022.2.0
PERSON2022.2.0

Additional tables covered by this module:

  • BULKMAIL, BULKMAILRECIPIENT, BULKMAILTESTRECIPIENT
  • MAIL_RUN, MAIL_LOG
  • CAMPAIGN, CAMPAIGNSTEP
  • EVENTDEFINITION, EVENT, EVENTPARTICIPANT, EVENTSLOT
  • DISTRICT, DISTRICTRESPONSIBLE

See CHANGELOG.md for the release in which each set was added.

Manual setup (without xRM 2022.2.0 preset)​

For projects on older xRM versions, wire AditoFaker up manually:

  1. Fulfil the prerequisites.

  2. Add the dependency to package.json:

    "devDependencies": {
      "@aditosoftware/adito-faker": "^1.1.0"
    }

  3. Create the folder structure under tools/AditoFaker/:

    tools/
      AditoFaker/
        configurations/
          configuration.json
          tableDefinition.json
        scripts/
          AditoFaker.js
          TableDefinition_generateCsv.js
          TableDefinition_syncFromAod.js
          TableDefinition_syncFromCsv.js

  4. Copy the current tableDefinition.json (and optionally configuration.json) from a recent xRM version into tools/AditoFaker/configurations/.

  5. Copy the four scripts from a recent xRM version into tools/AditoFaker/scripts/. Each script is a thin wrapper — for reference:

    // AditoFaker.js
    const { FakerAdito } = require("@aditosoftware/adito-faker");
    const fakerObj = new FakerAdito("./tools/AditoFaker/configurations/configuration.json");
    fakerObj.start();
    process.on('SIGINT', () => fakerObj.exit());
    // TableDefinition_syncFromAod.js
    const { TableDefinitionUtils } = require("@aditosoftware/adito-faker");
    TableDefinitionUtils.loadFromAOD("./tools/AditoFaker/configurations/configuration.json");
    // TableDefinition_generateCsv.js
    const { TableDefinitionUtils } = require("@aditosoftware/adito-faker");
    TableDefinitionUtils.generateCSV("./tools/AditoFaker/configurations/configuration.json");
    // TableDefinition_syncFromCsv.js
    const { TableDefinitionUtils } = require("@aditosoftware/adito-faker");
    TableDefinitionUtils.loadFromCSV("./tools/AditoFaker/configurations/configuration.json");

  6. Register the scripts in package.json:

    "scripts": {
      "AditoFaker-generateData": "node ./tools/AditoFaker/scripts/AditoFaker.js",
      "AditoFaker-generateCsv":  "node ./tools/AditoFaker/scripts/TableDefinition_generateCsv.js",
      "AditoFaker-syncFromCsv":  "node ./tools/AditoFaker/scripts/TableDefinition_syncFromCsv.js",
      "AditoFaker-syncFromAod":  "node ./tools/AditoFaker/scripts/TableDefinition_syncFromAod.js"
    }

  7. Run npm install. Continue as described in Installation.