Energy

Oil & Natural Gas Analytics

Optimizing oil & gas trading strategies with a unified data lakehouse.

Client: Fortune 500 petroleum, gas & energy conglomerate with operations in 100+ countries

The Problem

A Malaysia-based Fortune 500 energy conglomerate trades LNG internationally via cargo, term and spot trades. With trades booked across multiple ERP systems (SAP and TRMS), postings didn't match, no single system showed the exact trade information, and the finance team had no consolidated view of trade finances and P&L.

Challenges

  • No single source of truth — trades existed in SAP but not TRMS, and vice-versa
  • Manual reconciliation with finance spreadsheets to capture all trades
  • Very high accuracy expectations from the finance department
  • Every department ran its own analytics tools and frameworks — no single standard
  • Data redundancy and mismatches causing interdepartmental confusion

Solution Design & Implementation

01

Data Lakehouse Architecture

All source data loaded into a single location in its native form, analyzable on a single platform.

02

Data Reconciliation & DQF

A single source of truth with clean, reconciled data in the silver zone — reconciliation automated with PySpark notebooks.

03

Databricks-Powered Transformations

Bronze → silver reconciliation and data quality framework, plus silver → gold transformations on Azure Databricks.

Architecture

Sources

SAP

Invoice Data

SQL

Trading System

CSV

Shipping / Cargo Data

Excel

Finance / Planning Data

Ingestion & Integration

Ingestion via Azure Data Factory

Data Lakehouse & ELT

Azure Cloud Platform

Transformations

Databricks

Apache Spark

PySpark Notebooks

Data Zones

Bronze
Silver
Gold

Additional Stores

Azure Data Lake Storage

Delta Lake

Consumption

Power BI Dashboards

Orchestration & Monitoring

Apache Airflow

Workflow Orchestration Manager

Results

  • Reconciliations fully automated — no more manual cross-checking
  • Accurate, consolidated trade & P&L dashboards for finance
  • Reduced infrastructure costs
  • Significant savings on human capital

Future Plans

The DQF Project

Data quality algorithms extracted into a reusable Data Quality Framework for other products and applications.

Source Data Audits

Ongoing audits to keep source data correct for quality reporting.

Data Literacy

Lakehouse adoption to replace manual reporting with automated reports.

Data Governance

Organization-wide governance policies for security, standardization and compliance.