pyHQL (ASAM ODS Toolbox for Python) enables analysts and engineers to utilize the HighQSoft Query Language (HQL) to work directly with ASAM ODS data from Python, providing typed results, mass-data access with units, transactions, and file streaming for analysis, scripts, and dashboards.

HighQSoft is introducing pyHQL, a lightweight Python library that connects to the HQL WEB Service and, through it, to any ASAM ODS server in your infrastructure. You use familiar HQL commands to query metadata and measurement data. pyHQL maps ASAM ODS datatypes to convenient Python objects, allowing you to analyze them immediately.

pyHQL eliminates the effort of crafting low-level REST calls and interpreting ASAM ODS datatypes by providing a ready-to-use library—so teams can spend time analyzing data and focusing on insights rather than integration work.

For End-users

• Query information easily — Use the same HQL queries you already know to retrieve any information from ASAM ODS servers - with all the comfort you'd expect!

• Work natively in Python & Data Frames — Stay in your comfort zone with Python, Pandas, and friends.

• Transactions & file streaming — Open/commit/abort transactions from Python; upload and download files via dedicated helpers. Uncommitted transactions are safely aborted on disconnect.

For Analysts & Data Scientists

• Create dashboards — Build dashboards on ASAM ODS data with Plotly or other tools on top of typed results.
• Write helper scripts — Quickly retrieve and prepare ASAM ODS data for engineering teams or process information for other stakeholders.
• Automate pipelines — Combine pyHQL capabilities with Merlin’s pre-built stages to retrieve and write information from ASAM ODS and implement automated processing pipelines.

For Administrators

• Product Stability — Built on the well-maintained HQL WEB Service for consistent, predictable behavior.
• Simplicity — Get users started fast with a straightforward, two-URI connection model.
• Compatibility — Works with ASAM ODS 5 and ASAM ODS 6.

pyHQL is available now as a ZIP for pip install from the HighQSoft download page. The manual includes installation steps, connection examples, and a quick tour of queries, mass data access, transactions, and file transfer.

#ASAM #ODS #TestData #Merlin #Python #pyHQL #TDM #TestDataManagement #ASAMODS #HighQSoft

ASAMCommander is a modular web application for ASAM ODS–based test data management systems, providing engineers, analysts, and administrators with a customizable platform. It offers flexible data exploration and visualization, convenient ODS client, Matlab and Python access, and server-side analysis automation.

With ASAMCommander 25.2, we are releasing a major update to the AC-Mainframe and platform foundation. The new version focuses on corporate identity, embedded applications, user and group integration, and more predictable deployments. Platform owners and administrators can now brand ASAMCommander to match their environment, integrate more tightly with existing corporate authentication systems, such as OAuth, and operate external modules in a controlled and configurable manner.

In many ASAM ODS environments, ASAMCommander is more than just a client – it is the central entry point into a larger measurement data ecosystem. If branding, user management, or deployment patterns do not align with corporate standards, adoption suffers and operational overhead increases. The 25.2 mainframe update strengthens ASAMCommander as a platform, enabling it to be branded like any other enterprise application and supporting external applications through a consistent deployment and configuration model. This gives platform teams more control while reducing friction for end users.

Mainframe - Corporate Identity and Embedded Applications

• AC-Mainframe favicon, browser tab title, and logo can now be configured instead of hard-coded, allowing platform teams to align ASAMCommander with their corporate branding.
• Key-value properties from the configuration are merged with module defaults and made available to the embedded app. This allows environment-specific settings (such as backend URLs) to be implemented without requiring code changes.
• Improved handling of MF4 files, ensuring that large measurement files are not treated as generic attachments.
• A footer-based version display has been added and refined so administrators can quickly see which version is running without affecting the layout of embedded applications

Catalogs Management - User & group integration

• The user element has been extended with additional attributes to better align the AC user model with the directory.
• The importer for user groups has been updated to prevent changes in group structure from breaking the import in some corner cases.

Deployment & external application integration

• External applications can be attached to AC-Commander using, for example, a Helm-based deployment pattern, so custom modules can be deployed and configured alongside ASAMCommander
• A work package stabilizes the overall deployment by providing a clearer directory structure for both successful and failed runs.

Logging and language polish

• Logging configuration is more robust against minor formatting issues, helping to avoid missing log entries in job monitoring views.
• Wording and translations have been harmonized across modules (Navigator, Catalog, Search, Export, Bookmarks, Merlin-related views, etc.) for more consistent terminology in both English and German.

• Operate a branded ASAMCommander in production or for your customers - AC-Mainframe favicon, browser tab title, and logo can now be configured instead of hard-coded, allowing platform teams to align ASAMCommander with their corporate branding.
• Integrate with enterprise directories - Align ASAMCommander’s user and group model with an OAuth directory so that user attributes and group memberships stay in sync and imports remain robust over time.
• Deploy external modules easily - Use Helm-based configuration to attach additional applications to AC-Commander and pass environment-specific properties without hard-coding them into the code.
• Reduce operational friction - Benefit from clearer job directory structures, better handling of MF4 files, and many small UX refinements that remove blockers discovered during intensive customer testing.

ASAMCommander 25.2 is available as an update for existing ASAMCommander installations. Customers can obtain the release via the usual delivery channels or on request. For installation support, upgrade planning, or questions about compatibility with your ASAM ODS environment, please contact your HighQSoft representative.

#ASAM #ODS #TestData #ASAMCOmmander #TDM #TestDataManagement #ASAMODS #HighQSoft

HighQSoft is introducing the Merlin UI, a new module for ASAMCommander that brings server-side analysis and job orchestration right into the end-user’s web application. Engineers can discover data, launch analyses, and monitor pipelines end-to-end—without leaving ASAMCommander.

Why it matters

As data volumes grow, manual, desktop-based analysis tools don’t scale. Merlin is a second-generation, technology-independent framework that standardizes execution so teams get faster, comparable, and reproducible results.

What's new?

The following features are introduced:

For End-users

• Start Job toolbar action available across multiple modules to launch analyses from within ASAMCommander.

• Job bookmarks to quickly re‑run jobs or review prior results.

For business admins

• Template Manager to control which job templates are available to end users.

• Template Builder (Code & Graphical) to design and operationalize evaluations.

• Job Monitoring & Logging for live status, history, and traceability.

• Data Lineage to trace software, processes, and all interactions with your datasets.

What you can do with it

ASAMCommander is a modular ASAM ODS web application for data exploration and management. Out of the box, ASAMCommander already integrates expert tools like AVL Concerto, MATLAB, and NI DIAdem, and supports exporting ATFx, CSV, and MDF4—so you can keep preferred workflows in one place.

Merlin extends this with:

• Technology-independent executors (Python, Java, MATLAB, Spark)

• Resource management, queuing/scheduling, and event triggers (e.g., on import)

• Standardized, comparable results with full data lineage

Template types supported include importing, data validation, processing/analysis, statistics generation, archiving, and maintenance tasks.

About Merlin/Sam Analysis Server

Merlin is a second-generation, technology-independent framework for server-side analyses that orchestrates jobs, schedules execution, and scales across runtimes. It brings the algorithm to the data and ensures reproducible results with data lineage. Merlin integrates seamlessly with ASAM ODS-based test data management.

#ASAM #ODS #TestData #Merlin #DataLineage #TDM #TestDataManagement #ASAMODS #HighQSoft

ASAM e.V. published a new version of the ASAM MDF standard - an ideal container file as input for ASAM ODS-based Test Data Management systems. ASAM MDF is a binary measurement file format that was first released in 1991. The two major versions in use are Version 3.x (2002-2009) and Version 4.x (2009-). The standard is backwards and forwards compatible (within a major version).

The following sections provide an overview of major MDF 4.3 features.

Dynamic Data

MDF is usually used to store data records of fixed size. For a sorted file (only one channel group per data group), this means one can jump to any desired record when reading. There is also support for storing signals of varying size e.g. strings. Each string may have a different length, so the variable length data is stored separately (e.g. in an SDBLOCK) and referenced from the channel block. Here, the fixed record stores an offset used to locate the string in the SDBLOCK.

MDF also supports arrays of fixed size (including multidimensional arrays) where each element is the same type of data. These strategies cannot handle more complex cases, where the number (or type) of signals changes with each record. This is dynamic data.

There are two main motivations for recording dynamic data:

• Recording an existing serialization protocol (e.g. SOME/IP, Protobuf) as-is.
• Recording events where the number of signals is strongly variable in time (e.g. RADAR/ LiDAR).

MDF 4.3 introduces the DSBLOCK that can be used to mark a channel as containing dynamic data. In both cases, the raw data itself can be stored using the existing mechanisms for variable length data (VLSD, MLSD), fixed data or the new VLSC format.

The point of the DSBLOCK is to prevent readers supporting older versions of MDF from accessing and misinterpreting the dynamic data content, while referencing the description for MDF 4.3 compatible readers.

Standardized MDF Content Descriptions

Features in Autonomous Driving or ADAS (advanced driver assistance system) are dependent on measurements of the environment made by sensors. A single item recorded by a sensor (e.g. a snapshot at a moment in time) is usually called a frame.

MDF extends the concept of a ‘frame’ to a sensor event: both the raw frame data and additional properties needed for interpretation (e.g. image dimensions).

MDF 4.x:

• Bus Logging (CAN, LIN, FlexRay, MOST, Ethernet)
• Classification Results

MDF 4.3

• Raw Sensor Logging (Video, RADAR and LiDAR)
• GNSS (Global Navigation Satellite System)
• SOC (Service-Orientated Communication, SOME/IP)

GNSS Data Storage

GNSS (Global Navigation Satellite System) is the generalisation of GPS. Previous versions of MDF didn’t provide a way to unambiguously recognize channels containing GNSS data. This made it difficult for readers to correctly interpret GNSS data (e.g. routes with latitude and longitude) in files written by third-party tools.

SOC (Service Orientated Communication)

The new associated standard (Data Logging of Service Oriented Communication) is specifically designed to specify how SOME/IP is stored in MDF.

SOME/IP is a format for control messages sent over ethernet, used in AUTOSAR. It focuses on allowing clients to discover and subscribe to ‘services’ provided by a server. For example, this can be used to allow displays and telematics units to access data from cameras and other devices.

The standard defines names, tags and channel structures to allow a data logger to be able to store the raw content of the SOC data, with the important parts of the message structure ‘annotated’ in the form of MDF channels and metadata.

VLSC (Variable Length Data with Size Channel)

Most of the new dynamic data features can be stored in MDF using the existing mechanism for variable length signal data (VLSD) or maximum length signal data (MLSD).

However, VLSD has the disadvantage that the length of the variable signal data is always with the data itself, even after the file is sorted for efficient reading. This is often redundant when storing protocol data, since protocols often include the ‚count‘ as an additional channel.

VLSC allows the variable channel to reference a separate MDF channel that determines the count.

The changes are anticipated in the ASAM ODS 6.3 release which is scheduled for 2026.

#ASAMMDF #TDM #TestDataManagement #ASAMODS #HighQSoft

23 participants from 9 companies in 4 countries meet to test ASAM ODS client applications, ASAM ODS servers, and ATFX files for compliance with the ASAM ODS standard.

As in previous cross-tests, the participating companies came together to test and improve their software in real-life scenarios. The participants helped each other to understand the small differences in the data formats of the various programs and to solve the difficulties that can arise when interpreting this data. They were also able to discuss ambiguities and contradictions in the standard documentation, which can then be eliminated in the next version of the standard.

Once all the servers in the network had been set up, the first client connections were established. At the end of the first day, all clients were able to connect to all servers.
Each participating client was assigned an Excel sheet with the scenarios to be tested for each server. This time, the participants agreed to test only the http interface of the ODS standard. The scenarios were, for example:

• Opening the session
• Reading context variables
• Browsing through the test hierarchy
• Reading the model including the n to m relations
• Simple and nested queries
• Reading attributes and mass data including size and unit references
• Writing new instances and mass data
• Handling of external files (AoFile)

The tests were then carried out on the second day. Groups were formed with at least one representative each from the client manufacturer and the server manufacturer. A few identified problems were posted directly as issues in the ASAM GitHub of the ASAM ODS working group to be resolved with the upcoming release.

Today, we released a number of improvements for our products

ASAMCommander: Catalog Editor

The release includes several enhancements for managing catalogs in ASAM ODS (Physical Dimensions, Units, Quantities, and groups). The  Catalog Editor provides comprehensive CRUD (Create, Read, Update, Delete) operations and additional features like transaction control, drag and drop functionality, and support for managing catalog elements and their instances.

HQL: Transaction Handling for Outer Sessions

This update allows creating transactions outside of HQL and requesting HQL to use them. This lifts the limitation of only one transaction per ASAM ODS session.

Merlin/SAM: New Validation Job Pipeline

A new Validation Job Pipeline runs custom Python scripts on imported data if certain criteria are met. The scripts can generate new parameters or measurements based on the data.

Merlin/SAM: Check New Job Pipeline for Filter Criteria

Introduces a Job Pipeline for checking newly imported measurements against filter criteria stored in the ASAM ODS database. The pipeline stages include applying filter criteria, copying processing results, and triggering validation.

Merlin/SAM : Adaptation to Python Repository

Implements a git storage for loading remote stage functions for Job Pipelines making it easier to manage and update scripts.

All updates enhance the functionality and flexibility of the Test Data Management Ecosystem providing robust tools for data validation, transaction handling, and catalog management.

#HighQSoft #Release #HQL #SoftwareUpdate #ASAMCommander #ASAMODS

What’s the Deal with ASAMCommander? 

Test Data Management can be a headache. Multiple proprietary formats, incompatible data structures, and a never-ending stream of new information can make it difficult, if not impossible, to turn raw data into actionable knowledge.
At HighQSoft, we specialize in wrangling your measurement data into something you can use today and in the future. We offer multiple products and solutions to help you organize and manage data, regardless of volume and format.

Our ASAMCommander is a test data management application based on the ASAM ODS standard (https://www.asam.net/standards/detail/ods/). As a modular web application, it comprises loosely coupled, functional units called modules that form a customizable experience for end-users, system and data administrators, or data analysts.

The basic version of ASAMCommander is an out-of-the-box application that provides all end-user use cases for data exploration (navigation, search, visualization), expert tool integration (e.g., AVL Concerto, Mathworks Matlab, NI DIAdem), and data exports (ASAM ATFx, CSV, ASAM MDF).

The Enterprise Edition of ASAMCommander is suited for Global Test Data Management solutions based on the AReS Libertas ODS6 Platform. System administration features include system logging, data lineage, and data retention. Our Merlin Analysis Framework provides all features for complete automation of analysis pipelines. ASAMCommander Enterprise allows customization, and new modules, features, and workflows are easy to add or integrate.

ASAMCommander 24.2: Enhanced Features and Improved User Experience

The ASAMCommander release 24.2 is now available at our download portal and includes:

AC-Mainframe

• OAuth Authentication
  Support for OAuth authentication.
• Icon Size Fix
  Ensured consistent icon sizes across all (module) cards.
• Bug Fix
  An issue with highlighting and removing background color after clicking on a card and dropdown options was resolved.

AC-Navigator

• Tree Search Fix
  The tree generated after ‘Search’ now closes appropriately, and the ‘load more’ button is only visible when more children are available.
• Attributes Request
  Added the functionality to request attributes for application elements and display them in the Node Overview.
• Copy to Clipboard
  Implemented a feature to copy attributes directly to the clipboard.
• ASAM ODS Attributes
  Back-end support was added to show ASAM ODS attributes when selecting an ASAM ODS element.

AC-Logger

• Increased Usability
  Developed UI to indicate the current page after searching or sorting in the logs table.

General Features

• Extended CSV Exporter
  Integrated the support to export multiple measurements when exporting from a higher logical level.

The release contains various enhancements and updates for AC-MoMa, AC-Bookmarks, AC-Search, and AC-Merlin.

ASAMCommander works with any data model (ISO Safety, ASAM NVH, custom models), all domains (battery, powertrain, vehicle, ...), and any ASAM ODS server (AReS ODS Server, AVL Santorin).

#HighQSoft #TestDataManagement #MeasuredDataManagement #ASAMODS #ASAM #Release #Innovation

Libertas 24.1.3 brings a series of significant improvements and new features designed to enhance your experience. Here are the highlights of this release:

• Bearer Token Authentication: In addition to Basic Authentication, users can now log in using Bearer tokens (JWT), providing more flexibility and security in accessing the system.
• String Length Compliance: Libertas now strictly enforces string length limits, aligning with industry standards to ensure data validity.
• New String Data Support: This release includes support for additional string data types, enhancing the system's capability to handle diverse data formats.
• UTF-8 Character Set Support: Full support for the UTF-8 character set is now available, ensuring proper handling and display of international characters.
• Automatic User Group Assignment: New users authenticated via OAuth can now be automatically assigned to a default user group, simplifying user management and access control.
• Consistent File Downloads: Downloaded files will now consistently use the application/octet-stream content type, following industry recommendations for better compatibility.

These enhancements and fixes are part of our ongoing commitment to improving Libertas, making it more robust, secure, and user-friendly.

#HighQSoft #SoftwareUpdate #Libertas #Innovation

We are pleased to announce the release of ASAM Commander 24.1, packed with enhancements geared towards boosting stability and enhancing user experience across various modules, including some design upgrades throughout the entire application.

Web Application Enhancements

• Commander: Now supports localization in both German and English, alongside fixes for reload/refresh functionality, and the addition of module highlighting in the Navbar for better navigation.
• Navigator: Has undergone improvements, addressing issues with the Export feature within the Navigator Tree, refining the instance editor, and rectifying data type inconsistencies.
• MoMa: Now includes Elapsed Time in its monitoring table, enhancing its functionality.
• Bookmarks: We've upgraded the Bookmarks feature with keyword highlighting, improving search clarity.
• Search: Fixed pagination issues within Advanced Search, ensuring smoother navigation through search results.
• Merlin: We've resolved issues related to the Run feature in the HQL Job Type, ensuring seamless operation.
• Logger: Say goodbye to hostname identification issues! We've enabled container identification by name, ensuring accurate logging and traceability.

#HighQSoft #SoftwareUpdate #DataManagement #ASAMCommander #Innovation

Libertas 23.2.2 introduces several key fixes and improvements aimed at enhancing the performance and stability of the platform. Here's a summary of the notable changes:

• Performance Improvements: The data-create endpoint for PostgreSQL now benefits from multi-value insert statements, resulting in significant speed enhancements, with observed import time reductions of up to 50% in test environments. Additionally, batch size configurations for both data-create and data-update endpoints have been introduced, providing flexibility and optimization options.
• Model Read and Data Update Fixes: The release resolves problems with model reading for non-super users and ensures correct behavior of the data-update endpoint when handling empty values.
• Database Connection Pooling: A production-ready database connection pool has been implemented to increase connection reuse and mitigate the risk of reaching maximum connection limits. Configuration options are available via Manatee settings.
• OSGi Shell for AReS: A new OSGi shell for AReS has been introduced under the 'ares' command, enabling management of Libertas connections and resources via CLI.
• Database Recovery and Session Timeout Configuration: Libertas runtimes now support automatic recovery after database restarts, minimizing service interruptions. Session timeout mechanisms can also be configured via 'config.xml' files.
• Enhanced MS SQL Server Support: Improved support for MS SQL Server includes better transaction and connection handling, handling of large parameter queries, and ensuring compatibility with MS SQL Server SQL dialect.
• Tagging Support: Libertas 23.2.2 introduces support for tagging via HTTP headers, facilitating integration with tools like Postman.

#Libertas #LibertasRelease #SoftwareUpdate #HighQSoft