ModelMapper - Our Importing Engine for all file types

The ModelMapper is used to develop flexible and rule-based data importers for custom file formats. Generated measurement data comes in various file formats, data streams, or from other databases. Measurements may contain data of a minute, an hour, a day, or even more than a month. Our tools help us with the challenge of standardizing your data in ODS – no matter what.

 

We got several options to cover most if not all variations of data imports. With our data import engine “ModelMapper” there is a tool for the development of importer software to describe custom file formats in the standardized ODS application model semantics. It supports the analysis, transformation, and copying of information between ODS environments and different application models on basis of configurable “rules”. The usage of the ModelMapper requires a library of rules, which map all input information of the source to the target (element, attribute, …). Features of the ModelMapper are:

  • Address any instances of the application model (write access)
  • Map information of instances and attributes
  • Zip and unzip files
  • Move files from a source folder into the database

Once the importer job is done, the Avalon notification can send notifications to inform users about their data or a trigger to start a process. Work can begin!

Product Sheet ModelMapper Version 2.0 Status RELEASED Date December 2020 ODS Version ASAM ODS 5.3.1
IIOP Gateway
File Support All files including
.MDF4
.TDMs,
ATFx
Java Oracle JDK 8 or 11
AdoptOpenJDK 8 or 11
Highlights Flexible Rules Engine
Customization via XML

Process

Use-Case Description

In an ideal workflow, the sensors from the testbench can send the measured values to the ASAM ODS server or to some sort of gateway that can translate messages from the sensors and convert them into ASAM ODS API calls. That means: after executing a test, all the measured values are stored in the server with the associated metadata, and this data is ready for being used by the technical personnel. If this is not the case, you will need an importer software.

 

In a typical workflow, however, the sensors from the testbench are not smart enough to communicate with an ODS server. After all, the manufacturer of the sensors cannot know what the business application model looks like. It is also likely that the computers connected to the sensors also do not have such capabilities. In practice, the measured values will be read through some physical communication protocol (e.g., serial, USB) and the measured values will be retrieved as binary or textual files. After the end of a test, there will be a collection of files with the channel measurements and the metadata that will add information about the performed test (start time, equipment being tested, parameters to reproduce the test, etc.).

 

The ASAM ODS database can be populated with test data/metadata by writing a program that reads a collection of input files and calls the server API (CORBA or HTTP) for storing the data in a structured way. This solution works fine, but it demands an initial upfront investment (developer/hours) for implementing this custom importer. If multiple application models are needed, a general-purpose importing utility that can be used for different models is a benefit. At some point, it is necessary to create own toolsets to assist the process of import measurement data.

At HighQSoft, one of our most common tasks is to assist our customers with importing custom files into their application models. For this reason, the Model Mapper framework (also known as MoMa) was developed to facilitate the development of custom importers. This framework handles the main recurring tasks (i.e., tasks that are independent of the application model) and leaves as minimum work as possible for the parts of the importer that are tailored to the targeted application model.

 

The MoMa framework reads a script that configures the importing process and calls a set of rules that define how data will be read from input files and how the data will be transformed and written in the database. The MoMa runtime, as configured by the import script will be responsible for:

  • Establishing a session to an ASAM ODS server, and managing this connection during the import process
  • Writing instances to the application elements and linking them in a structured way
  • Handling transactions, i.e., starting, committing, and aborting transactions
  • Extracting zip files, deleting temporary folders, uploading files in the server, and linking them to AoFile instances
  • Parsing common file extensions such as CSV, XLSX, MDM, or other.

Occasionally, it is necessary to deal with a new and unique file format. In this case, there is no existing rule that enables us to parse this specific file format. The solution to this problem is to implement a new rule using the public API of the MoMa framework. The new rule can be added as a plugin and no modification in the engine itself is needed. Moreover, new rules can be kept in a library of custom rules so it can be available if the same specific file format must be dealt with in the future.

Features

Data Sources

Our ModelMapper can import data from any file (regardless of the format), folder, or database. Typical inputs for the Importer are CSV (comma separated values) and MDF (Measurement Data Format) files, or other ASAM ODS servers such as the Santorin server (an ASAM ODS server implemented in hardware).



Landing Zone

A data package usually contains one or multiple generated test data files that may either taken from the source directly or a defined landing zone, e.g. a defined folder and structure. This can be customized to the restrictions and requirements given.
Once parsed, the data is prepared for the next step.

Data Mapping & Enrichment

Test data usually does not come with extensive metadata to describe where the test is coming from, who tested it and what the purpose of the test was. This information can be enhanced by the import process. Either, additional information is provided via an extra file, an API with connecting information or it can be added at a later point in time by a client application.

Test Data Validation

Next to reading and parsing information, one important step in data importing is data validation. The Importer is able to validate data against formal restrictions of the business model, validate attributes against catalogs or even single attribute ranges. A good example is a validation of whether certain values are within a defined range or whether a recorded channel provided valid values.

Structural Verification

With built-in verification capabilities, the ModelMapper serves as a gatekeeper to the database. When writing data to the database, the data is already validated and of sufficient quality to get imported. The final check is whether all information is available and corresponding entries can be written.

Keeping original Formats

Usually, data is kept in mixed-mode storage. If so, describing data (metadata) is stored in the database and channel data (mass data) is moved to a protected location. Depending on the requirement binary files can be kept (e.g. MDF4) or a new internal binary format can be used. Files in text format will not be used.
Once the job is done, the Avalon notification can send notifications to inform users about their data or a trigger to start a process.

Supported Formats

Successful Test Data Management is most efficient, when based on validated test data. Our Avalon ODS Server takes test data from multiple sources and locations and organizes it – no matter which format it inherits. The most common file formats are shown below. However, itemizing data is our specialty. We are capable of importing almost every data type.

ASAM ATFx

Our Avalon ODS Server is compatible with the ASAM ATFx file format. ATFx files are used for import, data exchange and export. Our Manatee Web uses ATFx as a standard export.

AVRO

Our ODS Server is compatible with the Avro file format as an input. We can import from CSV files but store channel data in a resource economizing way in binary on the server-side.

Based on ASAM ODS 6.1 specification, we can export Avro files from the server.

CSV

Our Avalon ODS Server is compatible with the CSV file format. We can import from CSV files but store channel data in a resource economizing way in binary on the server-side. CSV is a standard export format for our Manatee Web.

HTML

ur ODS Server is compatible with the HTML file format as an input. We can import from HTML files but store channel data in a resource economizing way in binary on the server-side.

ISO MME

Our Avalon ODS Server is compatible with the ISO MME data format. The format can be easily imported.

JSON

Our ODS Server is compatible with the JSON file format as an input. We can import from CSV files but store channel data in a resource economizing way in binary on the server-side.

Based on ASAM ODS 6.1 specification, we can export JSON files from the database.

ASAM MDF 4.2

Our Avalon ODS Server is compatible with the ASAM MDF 4.2 data format – as well as all previous versions. ASAM MDF is a widely utilized as a container file that is easily managed with Avalon. With version 4.2 the format is also optimized for reading.

Parquet

Our ODS Server is compatible with the Parquet file format as an input. We can import from Parquet files but store channel data in a resource economizing way in binary on the server-side.

Based on ASAM ODS 6.1 specification, we can export Parquet files from the database.

National Instruments TDMS

Our Avalon ODS Server is compatible with the DIAdem TDMS Data File format. Often, we use DIAdem as a data import tool. Especially for imports from many sources, DIAdem is an essential tool to validate and consolidate data for our ModelMapper importer. Complex algorithms such as: calculated channels can be added prior to importing data.

txt

Our ODS Server is compatible with the TXT file format as an input. We can import from TXT files but store channel data in a resource economizing way in binary on the server-side.

xlsx

Our ODS Server is compatible with the XLS file format as an input. We can import from XLS files but store channel data in a resource economizing way in binary on the server-side.

xml

Our ODS Server is compatible with the XML file format as an input. We can import from XML files but store channel data in a resource economizing way in binary on the server-side.

any

Missing a format you would like to import into your test data management solution? We are sure we can help? Most formats are compatible.

HighQSoft GmbH

Black-und-Decker-Straße 17c
D-65510 Idstein