Contact us
data integration case
#ECommerce Development #Management Systems

Data migration and integration project for a big ecommerce enterprise

The customer approached us with the request to plan out, implement, and fully manage the process of data migration from multiple data sources into an integrated and secure location. Forbytes audited the client’s challenges and conducted data migration into the unified serverless warehouse while ensuring the data from various channels can be manipulated and exposed to insightful BI systems.

Microsoft .NET
Download
Read on

Scope of the project

1. Identifying data sources and data types

Our client is a big ecommerce group comprising more than 150 online stores and brands. Daily, each of their stores produces a tremendous volume of data. Plus, the company uses lots of back-office systems for managing their operations. As a result, the client gets a number of independent data sources, which makes it hard for them to make data-driven decisions.

The main data sources used by the client are databases, PIM systems, ERP systems, Google Analytics, Google AdWords, tracking systems, performance testing systems, Excel (Google Spreadsheets), etc. The key data types include:

data migration case study

2. Planning out and implementing the solution

The client’s request was to migrate the crucial historical data to Google’s warehouse and enable real-time/regular data collection across the main channels. The problem with disintegrated data management was that the client could view metrics on each store separately, without seeing the whole picture of their 150+ stores, business systems, plus back-office tools.

The company wanted to enable consistent data collection in one place and get role-based access to different datasets. The task was complicated by the fact that each dataset was managed and controlled by a different stakeholder within the enterprise. Our goal was to bring it all together and set an effective mechanism for data storage and collection.

data migration case

3. Exploring the results

We’ve managed to deliver measurable project results. Firstly, our team migrated data from the main sources, including Google Analytics, to BigQuery. Secondly, we ensured that these data are connected and can be leveraged in different types of reports. Both historical and real-time data are now available to the client.

Thirdly, we securely store the data in one place and expose them to powerful BI and analytics tools. Depending on the client’s purpose, the team can access different datasets and manipulate them in a way that drives useful business insights.

data migration and integration project

Their Challenges

Our Solutions

Data collected and stored in multiple locations were fragmented and hard to manage. The client requested us to migrate them to the all-in-one data warehouse to streamline management.

Forbytes took data from all the sources listed by the client and configured automatic data streaming in real-time while making sure that the data from different sources can be connected and used for report generation.

1

The client’s company generated and collected big volumes of data that had no practical value because they lacked control over their data across all channels.

As a result of data migration, the company got a role-based data management space where they can load, store, and prepare data depending on the focus of analysis and client goals.

2

Our client struggled with limited analysis opportunities and a tough decision-making process because there was no unified place to store, manage, and leverage their business data used for insight generation.

Our project team helped the client to extract value from data by connecting the data storage to insightful BI tools that facilitate data-driven decision-making and expand analysis opportunities.

3

Google Analytics 4 is replacing Universal Analytics in June 2023. The client was afraid of losing all their data stored in Universal Analytics. To securely preserve them, their team needed to migrate all the historical data to one location.

To prevent data loss and prepare the business to using next-gen Google Analytics 4, our team migrated all the historical data to Google BigQuery, which is a part of Google’s ecosystem optimized for work with vast databases.

4

Their Challenges

Data collected and stored in multiple locations were fragmented and hard to manage. The client requested us to migrate them to the all-in-one data warehouse to streamline management.

1

The client’s company generated and collected big volumes of data that had no practical value because they lacked control over their data across all channels.

2

Our client struggled with limited analysis opportunities and a tough decision-making process because there was no unified place to store, manage, and leverage their business data used for insight generation.

3

Google Analytics 4 is replacing Universal Analytics in June 2023. The client was afraid of losing all their data stored in Universal Analytics. To securely preserve them, their team needed to migrate all the historical data to one location.

4

Our Solutions

Forbytes took data from all the sources listed by the client and configured automatic data streaming in real-time while making sure that the data from different sources can be connected and used for report generation.

1

As a result of data migration, the company got a role-based data management space where they can load, store, and prepare data depending on the focus of analysis and client goals.

2

Our project team helped the client to extract value from data by connecting the data storage to insightful BI tools that facilitate data-driven decision-making and expand analysis opportunities.

3

To prevent data loss and prepare the business to using next-gen Google Analytics 4, our team migrated all the historical data to Google BigQuery, which is a part of Google’s ecosystem optimized for work with vast databases.

4

Technologies we have used in this project

Backend

.NET Core
ASP.NET Core MVC

Frontend

ASP.NET Core MVC

Database

MS SQL

Cloud Storage

Google BigQuery
Google Storage

Cloud Platform

Google Cloud

Continuous integration

Bitbucket

Log/Monitoring tools

Seri log
Custom logging

Cloud orchestration

Google Kubernetes Engine (GKE)

Virtualization

Docker

Other tools

Quartz

How We Add Value

01.Customizable solution

When dealing with a business challenge like the one faced by our client, companies have 2 strategies to choose from. They can build a custom solution or use off-the-shelf tools for data integration.

The problem with the second option lies in the inability of third-party service providers to fully meet customer needs. Off-the-shelf tools are capable of data migration from the most popular services. Apart from the popular systems, our client uses numerous custom solutions. Altogether, they will be expensive and hard to connect with a third-party provider.

After choosing Forbytes as a software engineering partner, the client got a customizable solution for data management and synchronization. The company now can add custom fields to the database, change the setup, modify data configurations, create custom reports, and more.

02.Flexibility in response to client needs

Off-the-shelf solutions could not provide the client with enough flexibility for cross-channel data management and integration. Our custom solution allows the company to adjust the data management process to their needs.

The data are stored, updated (either daily or in real-time), and managed in a unified warehouse. Some part of the raw data is transformed into insightful reports to expose them hereafter to BI systems. The rest of the raw data are stored securely and available for the client’s team 24/7.

All the data can be used for customizations and report generation. As per the client’s request, we can leverage any piece of data in reporting and make it a valuable source of insights for the client. What’s more, the flexibility of this custom solution enables the client’s BI team to customize reporting without the involvement of the tech team.

03.End-to-end services

The project was launched in the shortest terms, and we quickly started bringing the client’s expectations to life. Our team consisting of a software developer and a project manager took full responsibility for the project’s progress.

We organized and implemented the data migration process from start to finish. The Forbytes experts took initiative in fixing the unexpected issues to prevent them from damaging the project.

For example, at some point, we detected a few performance gaps and limitations of Google. The team took them into account and used our tech expertise to compensate for Google’s gaps with the stable data-centered processes in our system.

04.Automation

One of the client’s goals was to automate the processes that were done manually and required a lot of time and effort from their team. Prior to the project launch, it usually took 2 weeks or more for a manager to collect data from just a few sources in one place and create a report.

As for now, many business processes of the company are automated. Namely, the client got an automated data extraction process to keep data in one place.

The report creation process was automated as well. There are numerous static template reports generated regularly or as per the client’s request. Apart from this, their team has the option to create custom reports by connecting BI tools to the given datasets. Also, the team can generate custom-made reports that will be sent to the email or accessed via the web interface.

Future cooperation

We’ve successfully delivered the project’s goals and helped the client get a single source of truth for the main ecommerce channels. Now, the project is at the stage of support and maintenance. Our software engineer regularly monitors the data migration and integration processes and adds new customized reporting features as per the client’s wish.

One of the recent improvements to the solution was expanding financial analytics capabilities. The Forbytes team added the opportunity to include custom KPIs to financial reports. From time to time, we also get requests to enable automated integrations with new systems or add more reporting options.

Our cooperation with the client continues. The Forbytes team is involved in several sub-projects owned by the same company. We keep working on the client’s ecommerce solutions to prepare them for changing user demands and keep them performant and scalable.

data migration project

Contact us

Let’s discuss your project.

    Your personal data will be processed by Forbytes according to our Privacy and Cookies Policy.
    We respect your personal data and guarantee it’s safety.

    Forbytes has a strong process-oriented way of leading the company, with good respect for themselves and their managers. They made a choice not to become too big but build slowly on more qualified people, which is paying off.

    Niklas Hammar

    Niklas Hammar

    COO & Business
    Development at Offerta.se

    Forbytes has established strong project management. Their entire team is knowledgeable about appropriate processes and project expectations.

    Julia Noack

    Julia Noack

    eCommerce Manager
    Motorsweden & Swebike

    The Forbytes team has been loyal and dedicated in their partnership. The team provides management as needed but allows flexibility for direct work with developers.

    Thomas Andersson

    Thomas Andersson

    Ecommerce Manager & CTO
    Trademax

    Scroll Up

    This website is using cookies to give you the best experience.
    Continue using this site you agree with our Privacy and Cookies Policy.