data-migration-and-integration-project part of monitor
#ECommerce Development
#Management Systems

Data migration and integration project for a big ecommerce enterprise

The customer approached us with the request to plan out, implement, and fully manage the process of data migration from multiple data sources into an integrated and secure location. Forbytes audited the client’s challenges and conducted data migration into the unified serverless warehouse while ensuring the data from various channels can be manipulated and exposed to insightful BI systems.

Client:

eCommerce enterprise

Service type:

Data migration

Location:

Sweden

Technologies:

Microsoft .NET

Scope of the project

Our client is a big ecommerce group comprising more than 150 online stores and brands. Daily, each of their stores produces a tremendous volume of data. Plus, the company uses numerous back-office systems for managing operations. As a result, the client works with a number of independent and disintegrated data sources, including databases, PIM systems, ERP systems, Google Analytics, Google AdWords, tracking systems, performance testing systems, Excel (Google Spreadsheets), etc.

Key data types the client deals with

The problem with disintegrated data management was that the client could view data on each store separately, without seeing the whole picture of their 150+ stores, business systems, plus back-office tools. The company wanted to enable consistent data collection in one place and get role-based access to different datasets. The task was complicated by the fact that each dataset was managed and controlled by a different stakeholder within the enterprise. Our goal was to bring it all together and set an effective mechanism for data storage and collection.

Out data migration plan

We’ve managed to deliver measurable project results. Firstly, our team migrated data from the main sources, including Google Analytics, to BigQuery. Secondly, we ensured that these data were connected and could be leveraged in different types of reports. Both historical and real-time data are now available to the client. Thirdly, we securely store the data in one place and expose them to powerful BI and analytics tools. Depending on the client’s purpose, the team can access different datasets and manipulate them in a way that drives useful business insights.

Results

Challenge 1

Data collected and stored in multiple locations were fragmented and hard to manage. The client requested us to migrate them to the all-in-one data warehouse to streamline management.

Solution

Forbytes took data from all the sources listed by the client and configured automatic data streaming in real time while making sure that the data from different sources could be connected and used in reports.

Challenge 2

The client’s company generated and collected big volumes of data that had no practical value because they lacked control over their data across all channels.

Solution

As a result of data migration, the company got a role-based data management space where they can load, store, and prepare data depending on the focus of analysis and client goals.

Challenge 3

Our client struggled with limited opportunities and a tough decision-making process because there was no unified place to store, manage, and leverage their business data.

Solution

Our project team helped the client extract value from data by connecting the data storage to insightful BI tools that facilitate data-driven decision-making and drive insights.

Challenge 4

Google Analytics 4 is replacing Universal Analytics in June 2023. The client was afraid of losing all their data stored in Universal Analytics. To securely preserve data, their team needed to migrate them to one location.

Solution

To prevent data loss and prepare the business to using next-gen Google Analytics 4, our team migrated all the historical data to Google BigQuery, which is a part of Google’s ecosystem optimized for work with vast databases.

Technologies we have used in this project

Backend

.NET Core
ASP.NET Core MVC

Frontend

ASP.NET Core MVC

Database

MS SQL

Cloud Storage

Google BigQuery
Google Storage

Cloud Platform

Google Cloud

Cloud orchestration

Google Kubernetes Engine (GKE)

Virtualization

Docker

Log/Monitoring tools

Seri log
Custom logging

Continuous integration

Bitbucket

Other tools

Quartz

How We Add Value

Customizable solution

When dealing with a business challenge like the one faced by our client, companies have 2 strategies to choose from. They can build a custom solution or use off-the-shelf tools for data integration. The problem with the second option lies in the inability of third-party service providers to fully meet customer needs. Off-the-shelf tools are capable of data migration from the most popular services. Apart from the popular systems, our client uses numerous custom solutions. Altogether, they will be expensive and hard to connect with a third-party provider.

After choosing Forbytes as a software engineering partner, the client got a customizable solution for data management and synchronization. The company now can add custom fields to the database, change the setup, modify data configurations, create custom reports, and more.

Flexibility in response to client needs

Off-the-shelf solutions could not provide the client with enough flexibility for cross-channel data management and integration. Our custom solution allows the company to adjust the data management process to their needs. The data are stored, updated (either daily or in real-time), and managed in a unified warehouse. Some part of the raw data is leveraged in insightful reports and then exposed to BI systems. The rest of the raw data are stored securely and available for the client’s team 24/7.

All the data can be used for customizations and report generation. As per the client’s request, we can leverage any piece of data in reporting and make it a valuable source of insights for the client. What’s more, the flexibility of this custom solution enables the client’s BI team to customize reporting without the involvement of the tech team.

End-to-end services

The project was launched in the shortest terms, and we quickly started bringing the client’s ideas to life. Our team consisting of a software developer and a project manager took full responsibility for the project’s progress.

We organized and implemented the data migration process from start to finish. The Forbytes experts took the initiative in fixing the unexpected issues to prevent them from damaging the project. For example, at some point, we detected a few performance gaps and limitations of Google. The team took them into account and used our tech expertise to compensate for Google’s gaps with the stable processes running in our system.

Automation

One of the client’s goals was to automate the processes that were done manually and required a lot of time and effort from their team. Prior to the project launch, it usually took 2 weeks or more for a manager to collect data from just a few sources in one place and create a report. As for now, many business processes of the company are automated. Namely, the client got an automated data extraction process to keep data in one place.

The report creation process was automated as well. There are numerous static template reports generated regularly or as per the client’s request. Apart from this, their team has the option to create custom reports by connecting BI tools to the datasets. Also, the team can generate custom-made reports that will be sent via email or accessed via the web interface.

Let’s Get in Touch

    Size limit for file 5 MB (.doc, .xlsm, .pdf)

    Let’s Discuss Your Case

    Orast

    Orest Chaykivskyy

    Chief Commercial Officer
    Scroll Up

    This website is using cookies to give you the best experience.
    Continue using this site you agree with our Privacy and Cookies Policy.