NoWaste – Advanced BI Platform for Unified Warehouse and Operations Analytics
Nowaste, a leading European logistics company operating multiple warehouses, set out to turn inconsistent, siloed data from systems like WMS, HR, and ERP into a unified BI platform. The goal was to improve pricing strategies, reduce invoicing and reporting errors, and make data-driven decisions.
To achieve this, we built a data lakehouse using Medallion architecture on Azure Data Lake Gen2 and Databricks, complemented by advanced analytics and reporting through Power BI Service.
Key results:
100% data consistency for critical financial records
Eliminated errors and revenue loss due to unaccounted transactions
Insightful reporting and analytics for data-driven decisions
Context and Challenge
For over 17 years, our client has specialized in offering 3PL services to various industries across Europe. Since its foundation, they have been heavily investing in advanced automation of their warehouses becoming a disruptor in the market, especially with the rise of e-commerce. To maintain their high business standards and ensure their leadership remains unmatched, they required a modern analytics and BI system. This would enable them to deliver even more optimized and personalized services to their clients. As a highly technology-driven business, their numerous systems tracked nearly every aspect of warehouse operations, client service, and billing. However, over time, the massive volume of data became increasingly difficult to manage and analyze. The existing solutions they relied on were unable to effectively address the following challenges:- Scattered data: Key operational data was distributed across multiple systems, including WMSs, ERP, and HR systems, making unified reporting a cumbersome process.
- Complex invoicing: Ensuring each service was correctly billed required cross-referencing different systems, leading to potential errors.
- Need for advanced insights: To optimize operations and make data-driven decisions, the company needed high-level analytics and real-time dashboards.
Client’s Objectives
To solve the client’s issues, we set out to:- Data centralization: Create a single data lake environment that pulls, stores, and organizes all relevant data.
- Enhancing data quality: Improve data accuracy and consistency to avoid billing inaccuracies and potential revenue leakage.
- Enabling Advanced BI: Provide management and operational teams with user-friendly, insightful dashboards built on best-in-class analytics (Power BI).
Ready to Discuss Your Project?
We’re here to discuss your challenges and help you achieve your goals.
Solution Overview
After a thorough analysis of the client’s needs, we designed the following solution to address their challenges and provide a robust BI system:Incremental Data Ingestion
- We fetched data from all source systems and converted it into Parquet files, an open-source, column-oriented file format offering high-performance compression and encoding for efficient bulk data storage and retrieval.
- Our team established ETL pipelines using Databricks notebooks and orchestration to handle incremental data loads from each source system, ensuring optimal performance and scalability.
Data Lakehouse
Forbytes implemented the storage solution Medallion Architecture on Azure Data Lake Gen2 and Databricks, organizing data into:- Bronze layer: Raw data directly ingested from source systems. The purpose of this layer is to provide a historical archive of source data, data lineage, auditability, and reprocessing if needed.
- Silver layer: In this layer, the data is matched, merged, conformed, and cleansed so that the Silver layer can provide an “Enterprise view” of all its key business entities, concepts, and transactions.
- Gold layer: The Gold Layer contains business-ready, enriched data optimized for advanced analytics and reporting.
Materialized Views and Prepared Datasets
- Using dbt (industry standard for data transformation), most of the business logic, calculations, and transformations were executed within Databricks.
- We created materialized views and pre-aggregated datasets, reducing query complexity in Power BI and significantly enhancing dashboard performance.
Power BI for Advanced Reporting
- Our engineers developed advanced Power BI dashboards and reports to transform large-scale, complex datasets into actionable insights.
- Forbytes delivered operational metrics and financial dashboards tailored for invoicing, enabling the client to make data-driven decisions with ease.
Detailed Execution Timeline
This complex project required substantial involvement from business representatives, who acted as key interpreters of business flows, objects, statuses, and other definitions. Fortunately, our client had a clear vision of their needs, supported by comprehensive documentation, which enabled us to kickstart the project efficiently. Key project stages included:1. Requirements walk-through
Our team reviewed the client’s requirements in detail to ensure a complete understanding of their needs. This thorough analysis provided all the information necessary to start the project with confidence.2. Infrastructure and platform setup
Since the client was already using Azure, we built the solution leveraging Azure tools, including Azure DevOps for CI/CD and Azure Data Lake Gen2 for data storage, ensuring seamless integration with their existing ecosystem.3. Data capturing, transformation and storage
Data was updated daily by capturing changes from all source systems and storing them as Parquet files in the data lake. Using Databricks Jobs and Notebooks, raw data was processed and moved through the Medallion Architecture layers (Bronze, Silver, and Gold). Final datasets were prepared using dbt, transforming and storing them in the Gold Layer, ready for analytics and reporting.4. Power BI service and reports creation
Data visualization and reports were built using Power BI Service, leveraging the prepared datasets in the Gold Layer for easy and efficient reporting. We tailored reports based on type and purpose, with some shared directly with the client’s customers. Customers, impressed by the high-quality insights, placed additional orders for custom reports to address their specific needs. The project was completed in 30 weeks and continues to evolve, with ongoing efforts to add more complex reports for the client and build custom reports for their customers.Results and Impact
Quantitative Results- Data Consistency & Accuracy: Achieved a 98 100% consistency rate in financial records, reducing invoice disputes by 30%.
- Eliminated revenue loss Our client previously faced challenges in accurately accounting for all services and invoicing them. With the implementation of the new BI system, 100% of services are now accurately accounted for and billed.
- Faster Insights BI reports that previously took days to compile can now be generated on- demand in seconds.
- Reduced Manual Effort Automated data feeds saved each department 5 10 hours per week.
- Improved Confidence in Billing The finance team can accurately invoice clients, mitigating revenue leakage.
- Enhanced Decision-Making Operational managers can pinpoint inefficiencies in warehouse processes and address them promptly.
- Scalable Foundation The medallion-based lakehouse architecture supports future data sources and advanced analytics (e.g., predictive, ML-based insights).
Technology Stack
Azure Data Lake Gen2 – Scalable cloud-based data storage Databricks – Data engineering and transformation platform Medallion Architecture – Data layering approach (Bronze, Silver, Gold) Parquet – Columnar file format for efficient data storage and retrieval dbt (data build tool) – Data transformation and materialized views Power BI Service – Advanced analytics and reporting Azure DevOps – CI/CD pipelines and infrastructure management .NET – Part of the client’s wider tech ecosystemKey Takeaways and Lessons Learned
Success Factors- Robust Data Architecture Adopting the Medallion approach ensured data was refined incrementally, reducing errors early.
- Close Stakeholder Collaboration Frequent feedback loops with finance and operations minimized misaligned requirements.
- Security & Compliance Proactive focus on data access controls and encryption prevented governance issues.
- Complex Legacy Systems Some older WMS/ERP modules required custom connectors and data format adjustments.
- High Data Volume Optimizations in Databricks and careful partitioning in Azure Data Lake handled large daily data ingestions.
- Importance of Data In today’s market, staying competitive is impossible without using your data to continuously analyze opportunities for greater efficiency and effectiveness.
- Data Lakehouse Maturity Combining the best of data lakes and data warehouses (lakehouse) gives logistics companies a flexible yet structured approach to data.
- Business Intelligence Adoption Power BIʼs user-friendly interface accelerates analytics adoption across non-technical teams.
Want to Achieve Similar Results for Your Business?
We’re here to discuss your challenges and help you achieve your goals.
Our Focus Industries
Let's Get in Touch
Forbytes created an app that allowed users to quickly measure themselves to be fitted for custom shirts. As a result, we could increase our speed, quality, and shirt output per week. Forbytes was dedicated to delivering a solution that accurately addressed our needs.
Forbytes’ engineers and team leads worked as part of our RnD group and were directly managed by our company directors. They were both proactive and adaptable, qualities that perfectly match our needs in our fast-growing and constantly evolving environment.
Forbytes has a strong process-oriented way of leading the company, with good respect for themselves and their managers. They made a choice not to become too big but build slowly on more qualified people, which is paying off.
Niklas Hammar
COO & Business
Development at Offerta.se
Our e-commerce site saw online inquiries and orders rise after working with Forbytes. The team used top-notch expertise to inform their suggestions and improvements. Constant communication, quick responses, and prompt delivery made for a smooth and enjoyable collaboration.
The Forbytes team was very professional and demonstrated a great understanding of the business aspects. All tasks were completed above expectations. Their experts were always available, and we never needed to wait for a response or a suggested solution.
