At HighQSoft, we are offering the integration of big data technologies in multiple ways. These incorporate various strategies for scalability, storage technologies, and integration of analysis.
Our Plankton Ecosystem describes our microservices platform around the AReS ODS Servers. It includes several gateway products for data access, a dispatcher layer to adopt alternative and none-SQL databases, and provided extended services to fulfill the customer use-cases. The target of Plankton is to provide a business API to the customer that incorporates a backend infrastructure of choice - including technologies listed below.
With our AReS ODS server, we got a core product ready to start with. As part of our Plankton EcoSystem, we make its utilization versatile. We provide several gateway products for system access, are prepared for alternative data storage technologies, and data analytics with Apache Spark or Python. We are further developing the EcoSystem to provide a test data management platform that adapts to the business use case.
Big data is scalability. For scalability, we are using Docker containers and Kubernetes to manage them. Whether the system is integrated into a local data center or a cloud, containerization is a key element for future scalability. It also enables cloud readiness as most cloud services provide this option. Our software products fulfill this use case.
Big data integration is all about the use-case and the choice of technologies. Not all technologies fulfill all use-cases. With our AReS Server, Merlin Analysis Server, and HQL, we are prepared for requirements that derive from the assessment. By the end of 2021, we gathered 3 years of experience of running a full-fledged Test Data Management System with Event & KPI detection based on Apache Spark, Apache Hadoop, and ASAM ODS Parquet files.
ASAM ODS 6 has become much more flexible in administration, deployment, and scalability with its HTTP API. Our AReS Server is compatible with the HTTP API as well as modern containerization software such as Docker and Kubernetes. As part of the Plankton Ecosystem, our AReS Server includes a gateway for single-node access and is prepared to adapt to future data storage technologies. With that, the data storage may switch but the ODS API and/or the business gateway remains the same.
Today, we providing gateways for our products AReS ODS Server, HQL, as well as the IIOP Gateway. All are single node entry points for Test Data Managements systems and scale with increasing requirements. The gateways are based on our microservice Plankton platform and target to provide a single API for the enterprise. Customer-specific gateways, for example including ASAM ODS access with specific analysis functionality, are part of the plan. These we call business gateways - a platform solution for your enterprise.
As part of our Plankton Ecosystem, we are providing microservices-based services for the generation of JSON, Parquet, and AVRO files defined by ASAM ODS 6.1. The components are integrated within data import or based on the ASAM ODS Server notification services. Thus, test data can conveniently be exported to a Hadoop system for data analytics with alternative access APIs.
Some data requires alternative storage technologies. Especially indexers provide a set of features such as quick filtering, facetted search, and auto-complete search. We are incorporating Elastic products to fulfill these use-cases. Whether we store Event & KPI data in an indexer and use it directly from the web application or if it is a data source to AReS Server (Plankton), depends on the use-case
We integrated Apache SPARK as a processing engine for event and KPI calculations. Our Merlin Server combines measurement data with the customer algorithm and calculates the result. As we are independent of data models, the process and integration usually have the character of a product-based solution.
The ASAM ODS / BODS definitions of the file formats JSON, Parquet, and AVRO were developed for big data technology integration. Our event and KPI calculations are based on Apache SPARK, Hadoop, and Parquet. With our Plankton microservices and Merlin analysis framework, we are independent of the technology. The Hadoop Ecosystem is comprehensive in itself. We will propose a solution based on the customer use-case and requirements.