Real problems, real solutions.
A look at data engineering work we've done for past clients — what wasn't working, what we built instead, and what changed as a result. Client details are kept general to protect confidentiality.
Replacing a Fragile Overnight Database Backup with Direct, Incremental Loading
A regional HVAC & maintenance services company, migrating from SQL Server to Databricks
The Problem
This client was in the middle of moving their data platform from SQL Server to Databricks. Every night, their process had to back up entire databases, then download all of that data before it could be pushed into Databricks and Azure Data Lake Storage — even on nights when little or nothing had actually changed.
The backups alone took a long time. The full downloads took even longer. And if anything failed partway through — even near the very end — the entire process had to start over from scratch. When that happened, staff would arrive the next morning to find the data they needed simply wasn't available yet, sometimes for hours.
The Approach
Rather than trying to speed up the existing backup-and-download process, we looked upstream at where the data in SQL Server was actually coming from — and found that the original source already offered APIs capable of delivering the same data directly.
We rebuilt the pipeline around that discovery: Databricks notebooks now pull data straight from the source API into Azure Data Lake Storage as Parquet files, loading only new or changed records using Databricks Auto Loader, rather than the entire dataset every time. The SQL Server backup and download steps were removed from the process entirely. The Power BI reports built on top of this data were left untouched — same data, same reports, nothing for end users to relearn.
Nightly database backups eliminated entirely. Full data re-downloads replaced with incremental loading — only new or changed records move each run. Load time cut from hours down to a fraction of that, and a failed run no longer means starting over or leaving users without access to their data the next morning.
Giving an HR Team Live Visibility Into a Workday Rollout
A company migrating employee data into Workday, alongside their existing Microsoft Entra identity system
The Problem
This company was in the process of rolling out Workday as their new home for employee data, which meant every employee needed to be added and onboarded into it. The trouble was, there was no easy way to see who had already been added to Workday versus who still needed to be — the onboarding team had to manually cross-reference Workday against their existing Microsoft Entra identity system to figure out who was missing.
On top of that, the team just needed a simple, general way to look at their employee data across both systems while everyone was still getting comfortable with a brand-new platform.
The Approach
We pulled employee data directly from both the Workday and Microsoft Entra APIs into Databricks, then built materialized views on top of Delta tables to represent both platforms' data side by side, in a consistent, comparable shape.
On top of that foundation, we built an interactive Power BI dashboard the onboarding team could use day to day — instantly showing who was already set up in Workday versus who still needed onboarding, plus a broader, easy-to-read view of employee data across both systems to help bridge the learning curve of adopting Workday.
No more manually cross-referencing two systems. The onboarding team could see Workday rollout progress at a glance, and a new, easy-to-read view of employee data helped the whole team get comfortable with Workday faster.
Keeping Power BI Reports in Sync During a Live Fabric-to-Azure Databricks Migration
A growing company consolidating its data platform onto Azure Databricks to support an expanding AI/ML roadmap
The Problem
This client was consolidating onto Azure Databricks as their primary data platform, driven by company growth and an expanding set of AI and ML initiatives on their roadmap. That shift meant rebuilding the data sources behind their existing Power BI reports — data that used to live in Fabric was being recreated in Azure Databricks as views and Delta tables.
The trouble was, their Power BI reports and semantic models still lived in Fabric, built on top of that data. Without a way to tell Fabric the moment new data was actually ready in Databricks, refreshes had to be scheduled blindly — with no real way to know whether the underlying Databricks data load had finished successfully before Fabric tried to refresh from it. That risked reports quietly showing stale data throughout the migration.
The Approach
We built a pipeline that connects directly to Microsoft Fabric's API, letting Azure Databricks trigger semantic model and report refreshes itself the moment new data was actually ready — removing the guesswork of scheduled refreshes entirely, and keeping both platforms working together smoothly throughout the transition.
The pipeline follows a bronze/silver/gold structure: raw data lands in bronze, gets cleaned and processed in silver, and only fully-validated data reaches gold — which is what Fabric's semantic models are built on. The moment gold finishes for a given object, the pipeline calls Fabric's API to refresh exactly the reports tied to that data. To scale this across many workspaces without hardcoding each one, we built a single Delta table holding metadata on every Fabric workspace and object involved — the pipeline reads that metadata to decide what to process and refresh, rather than needing custom logic per object.
Reports refreshed automatically, only once data was actually ready — never on a blind schedule. A single metadata-driven pipeline handled every workspace and object involved, so new ones could be added without rebuilding the process each time, and reports never went stale during the migration.
Right-Sizing Power BI Access Through Automated Report Delivery
An organization aligning Power BI license spend with what each user group actually needed to see
The Problem
This client had a Power BI report set up so several different groups of users could each view it based on their needs. Once the report was in use, it became clear that these groups only ever needed to see one specific dashboard — not any of the report's other pages, and nothing else in Power BI at all.
Giving each of these users a full Power BI license to view just that one page would have meant paying for access far beyond what they actually used. The organization wanted a way to right-size that access — matching what people paid for to what they actually needed.
The Approach
We built a process that connects to the organization's Power BI instance in Microsoft Fabric and automatically exports the specific dashboard these users needed as a PDF.
That PDF is then emailed directly to each user group on a schedule — delivering exactly the data they needed straight to their inbox, with no Power BI license, login, or new application for them to navigate at all. They simply open an email, same as anything else they already receive.
Access matched to actual need, with no unnecessary licensing overhead. Users got exactly the data they needed, automatically, straight to their inbox — a simpler experience for them, and license spend that reflected real usage for the organization.
Have a similar migration or pipeline problem?
If part of your process sounds like the "before" picture above, let's talk about what the "after" could look like for you.
Get in Touch