Bugtracking License Activation ManateeWeb Demo
Big Data with Measurement Data 2017-04-19T11:06:16+00:00

Our Developments in the Big Data Domain – BIG ODS

Our customers demand a solution with integration of big data technologies. And rightly so.

Scalability of ASAM ODS Systems

linie_grau

Recently, a lot of key features were introduced to software solutions in order to address the scalability of an ODS system. We can address the demand for not restricting measurement data management services to solely one department or use case, as it is already state-of-the-art.

While Oracle remains as a single and local gateway for access, it surely owns the performance to manage the measurement meta data and access to the mass data very quickly. Our Avalon ODS Server always featured a parallel instantiation to e.g. channel workload from import processes, users and analysis.

With our Avalon Server Suite 2015 we introduced the Avalon Distributor. Now it is possible for all (or many) users to use an Avalon Gateway to balance the workload on multiple other Avalon ODS Servers. Moreover, you can use the distributors with fallback options, so your whole server infrastructure is save. Thus, ODS offers horizontal and vertical scalability in data management.

Within recent times, data access via web was commonly introduced into ODS. Hence, there is a solution to the main challenge of viewing, browsing, pulling or exporting and sharing the data of any data model. Web platform feature applications, such as the ManateeWeb application and the Manatee Integration Platform, provide data access from anywhere in the world. Additionally, the ASAM Web Services and the HighQSoft Query Language Libraries (HQL) enable developers and engineers to integrate their tools, platforms or analysis into ODS without greater efforts or ODS API know-how.

A rather new aspect to ODS is the integration of automated evaluations. Not only import processes may trigger evaluations to validate the incoming data, but also the ODS server itself to conclude some standard (e.g. fleet) analysis. The engineer himself can cause evaluations, too. The important fact is, that the analysis is brought to the data itself. This is not only a backbone to future big data integration, but also valuable for performance, comparability of results and data redundancy. An approach to analysis is always a solution. Our products Merlin Analysis Server for Java integration and our soon-to-be Matlab support via HQL are part of it.

All those aspects of a solution integration match the state of the art in ODS and have enough performance for most cases within the ASAM ODS domain.

Discussed technologies and approaches are very important for further development towards Big Data with measurement data, as current alternatives (big data homegrown) do not address the requirements of measurement data management properly.

News / Outlook / Visions: Performance and Scalability
by Dr. Ralf Nörenberg, HighQSoft GmbH
Outlook on HighQSoft GmbH activities from May 2015 to May 2016 with focus on big data subjects. Introduction to specific research projects.
Download: UGM2015-HQS-Scalability-and-Performance.pdf

 Why ODS Has to Go Big Data (BIG ODS)

linie_grau

Current ASAM ODS solutions target at retrieving data from a source to physically store them in a “mixed mode” system. This mode stores generated data two-fold: Meta data (e.g. measurement descriptions) are copied into a centralized relational data base (e.g. Oracle), whereas the actual mass data are moved onto a file server. The files are either generated files or the original ones. Both is possible. The content of the files is itemized and accessible by ODS to get only the information required if necessary. Furthermore, the content is only referenced. Hence, the files may be stored decentralized on multiple file servers.

The system solution described (see also “Scalability of ODS Systems”) works for most measurement data projects.

This system design works for most projects managing measurement data. Data access on meta data is indeed quickly established and has a great performance itself. Certain features like a “facetted search”, “search on mass data” or “on-demand analysis” are beyond limit for the ODS systems, though. But it is possible to find a remedy by integrating an indexing service into the solution. However, this approach is not standardized and implemented according to customer requirements.

Now, two new major requirements are brought up by state-of-the-art test engineering that cannot be implemented without a thoroughly planned big data system:

  • large scale data volumes from vehicle testing
  • demand for big data analytics feature (in ODS)

The large scale data volumes originate from vehicle testing, e.g. driver assistant system that capture the human senses within data. Plenty of data. We talk about 1000 vehicles with 100 measurements with 10^3 to 10^5 channels a day (2*10^10 / year) to cope with future requirements.

As none of the home-grown tools of the big data domain currently have an answer to the very strictly organized data of ASAM ODS and its use cases, utilizing its tools for benefits in (fleet) analysis is vital. The big data analysis part needs to become part of the ODS solution.

As an outlook, it will still be an ODS system, that organizes and manages the measurement data, as the automotive domain requires the benefits of comparability, long-term availability of the data and stability of the standard. The question is how to combine both worlds and utilize each of their benefits.

The ASAM ODS Big Data Proposal

linie_grau

In order to fulfill the new requirements of large scale data management and big data analysis, the domain and all its members formed an initiative to investigate and prototype a solution.

The investigation starts at the ODS API. Currently, a neutral partner is contracted to investigate new API protocols to adapt to state-of-the-art-technologies. This is done by one of the famous Fraunhofer Institutes in Germany within the ODS 6 proposal.

What is more, it is necessary to investigate the physical storage of data. This is done by the ASAM Big Data Proposal.

Arranged by Cummins in USA, the initiative has members like AUDI, Cummins, ETAS, Evolutionary Software, Ford, General Motors, HighQSoft, RDE datasolutions, White Pine Software Technologies and many more, in order to reach their defined target: A working prototype that integrates big data in physical storage and analysis. The ASAM e.V. receives the results of this project for standardization.

The major aspect of the integration is the question of how to address the multiple technologies that are available (mainly Hadoop-based) without leaving the standard behind / having a proprietary solution?

One valid approach, submitted by HighQSoft, is to define an abstract access layer as a middleware between ODS and the big data technologies. This helps to adapt to multiple data store solutions even within a cluster. Addressing multiple clusters also is not out of the picture. This way ODS remains to be the access level for management, and most importantly security. The data itself goes to a big data cluster that provides the data storage and performance scalability. ODS calls it, big data runs it.

We as HighQSoft are part of the proposal and intend to contribute largely to the development.

The HighQSoft Approach to Big Data

linie_grau

We are a main innovator and contributor as well as an ODS server provider to the ASAM ODS standard. Therefore, our current research activities towards prototyping an implementation are underway. In order to cope with the challenge of big data clusters, we confederated with big partners like Bertrandt and ETAS. Both companies own clusters and can provide data for testing.

The chosen approach is twofold: With having our Avalon ODS Server around, which already has a framework to integrate new drivers as a plug-in, we develop a driver for accessing the big data cluster. Not directly, though. We implement “big data web service” as a middleware in order to gain another level of standardization. The “big data web service” is part of the big data cluster. It retrieves all inquiries obtained by the Avalon server and transmitted through the driver.big data and asam ods - big ods for measurement data management

This way, the web service separates the ODS data management from the zoo of already existing big data technologies. Within the web service, the jobs to read and write ODS-compliant data need to be implemented. Jobs are generic in their task but specific to a technology, e.g. a MongoDB or Parquet implementation. However, it is possible to standardize the jobs, like the web service, in their definition. This allows to adapt to any scenario a customer potentially has in mind, when he sets up his own cluster.

To take this approach one step further, the big data web service will inherit a feature to integrate third-party (analysis) jobs to be managed by ODS and be executed within the cluster. This allows targeted evaluation of the measurement data according to the engineers’ demand.

The current development is underway and aims to identify definitions and to hand them over to the ASAM e.V. for standardization.

Big Data ODS
by HighQSoft GmbH
Outlook on the HighQSoft approach to integrate big data into ODS to make it BIG ODS.
Download: Big-Data-ODS.pdf

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close