Customer
The Customer is a conglomerate with multi-million revenue that runs 15+ diverse businesses and deals with over 200 world-famous brands. The company operates in such areas as engineering, logistics & inventory, automotive industry financial consulting (for banks, investment companies, etc.), retail, food & beverage, advertising and more.
Challenge
The Customer’s businesses had independent IT infrastructures. Thus, operational data was isolated across multiple CRMs, ERPs, POSs and ecommerce solutions, supply chain management systems and other enterprise systems. As a result of disintegrated data, the Customer encountered a number of inconveniences:
- Manual data transferring led to errors and hindered data reliability. Moreover, the data wasn’t available immediately as manual inputs were bound to employee schedule.
- The employees couldn’t quickly get a complete picture of all customer interactions with the business.
- There was no ability to implement an advanced conglomerate-wide loyalty program.
- Automatized reporting embracing all business directions was impossible.
Solution
Assessing the existing needs and problems
Our consulting team started with a thorough examination of the Customer’s applications. During several Q&A sessions with the senior management of each business, our consultants identified existing business processes, information flows, technical details of communication protocols, data formats in use and the required data refresh rates.
Having assessed the revealed problems in the examined processes, Our IT consultants suggested integrating the applications and systems in use to make the Customer’s IT infrastructure better serve the current business needs.
The integration solution had to unite dozens of enterprise software components from 6 industries of different natures. Among them were several CRMs, several ERPs, web and mobile ecommerce solutions, multiple point-of-sale systems (POSs), supply chain management systems, several databases, reporting and analytics tools and more.
The integration architecture pattern had to connect applications placed both on-premises and in the cloud, and support different data formats (text, media, etc.). It was required to handle sync and async messaging, batch processing and support RFC (Remote Function Call) communication, while ensuring the IT infrastructure’s high performance and preserving its flexibility.
Developing application integration architecture
The team of Our IT consultants offered to implement a layered architecture. The suggested architecture pattern included 4 layers, each fulfilling its specific role and functions:
- Points of initial data entry (included CRMs, ERPs and other enterprises systems of different purposes).
- The integration layer (mediator) – responsible for data extraction, transformation into the needed format, enhancement, and routing. It was decided to extract data directly from source applications since data processing in the data warehouses adversely affected the data veracity and the process velocity.
- Single and consistent data storage (with thought-out data access and encryption) – responsible for storage and processing of operational and analytical data.
- The Business Intelligence (BI) layer – responsible for data analytics, reporting and visualization.
The layered architecture was chosen as it enabled:
- Isolate future changes within one layer.
- Allow functionality re-use (from the lower layers).
- Eliminate security concerns.
- Make the conglomerate-wide IT system more understandable, simplify its testing and maintenance.
The proposed integration was based on ESB (enterprise service bus) and/or ETL (extract, transform, and load) services to allow for two strategies of smooth data exchange. ESB should be responsible for quick and relatively lightweight operations (messaging) between a data source and another data source or the underlying layer. ESB handled message routing, message exchange monitoring, message storage and message format validation.
ETL (extract, transform, and load) tool should be mostly responsible for loading massive amounts of data. ETL dealt with the transformation of structured and unstructured data into usable forms, data cleaning (by applying advanced validating rules), master data management and loading transformed data into the data warehouse.
Evaluating and comparing technology options
As a part of the IT consulting service, we described, assessed and compared four different tech stacks. The options arose from the Customer’s partnerships with IT providers and tech prevalence in the existing IT infrastructure. They included:
- Option 1 – integration based on the SAP tech stack and SAP Process Integration (ESB).
- Option 2 – integration based on the Microsoft tech stack and Confluent for Kafka / Azure Service Bus (ESB-centered).
- Option 3 – integration based on the Microsoft tech stack and Talend Data Fabric (ETL-centered).
- Option 4 – integration based on the Microsoft tech stack, Talend Data Fabric as ETL and Talend ESB.
In view of the Customer’s cloud migration strategy and plans to introduce IoT and social media streaming, the options included either cloud-ready solutions (Option 1 and 4) or those requiring minimal efforts for migration to the cloud (Option 2 and 3). ScienceSoft consultants highlighted the recommended architecture option for each way.
Reporting
Our IT consultants prepared comprehensive descriptions and assessments of each proposed integration scenario. They included the integration design and integration guidelines, data extracting & loading, data processing operations, data quality & master management, data storage structure. They also provided detailed recommendations on the overall infrastructure design, rated the strengths and weaknesses of each tech stack, listed available connection types. All the needed tools were compared against key parameters and expected costs of ownership.
In addition, we described the potential risks related to the layered architecture and came up with the mitigation plan for each of them.
As the integrated enterprise applications had to deal with sensitive personal and business data, our IT consultants also described data security strategies for each proposed integration pattern in the report.
Results
The Customer obtained four comprehensively described and assessed integration scenarios and tech stacks for the quick and effective consolidation of the conglomerate’s data. All four action plans allowed for having clean, accurate and consistent data, timely available and moved safely across different systems.
To support the company’s growth and scalability, the suggested integration option allowed for the smooth introduction of new components at minimal extra cost.
Methodologies
Q&A sessions, functional decomposition, business requirements analysis, comparative analysis.