Data Engineer

Title: Senior Technical Specialist - PBI Databricks

Location: Remote

Duration: 12+ Months

Education: Any Degree

Exp Level: 8+ Years

Job Responsibilities:

• Design and build conformed dimensions and transactional fact tables for enterprise reporting.

• Develop and maintain data pipelines on Azure Databricks using PySpark, Spark SQL, and Delta Lake.

• Implement best practices in dimensional modeling, including:

• Star schemas

• Conformed dimensions

• Surrogate and deterministic hash keys (e.g., SHA-256)

• Transactional fact design

• Translate existing SQL Server and stored procedure logic into efficient Spark-based transformations.

• Implement incremental data processing and change data capture (CDC) pipelines.

• Ensure data reconciliation with source systems to meet accuracy and audit standards.

• Collaborate closely with a senior Databricks lead to deliver core data assets within defined timelines.

Required Skills:

• 8+ years of experience in data engineering, with strong focus on Azure Databricks.

• Proficiency in:

• PySpark

• Spark SQL

• Delta Lake

• Deep expertise in dimensional modelling and data warehousing concepts.

• Experience with:

• Building scalable production pipelines

• Incremental data loads and CDC patterns

• Performance optimization in distributed systems

• Strong understanding of relational databases and ability to interpret complex SQL/stored procedures

Good to have

Experience with:

• Lakeflow / Spark Declarative Pipelines or dbt

• Unity Catalog

• Exposure to financial/General Ledger data models

• Familiarity with JD Edwards systems, including:

• F0902 (balances)

• F0911 (journal details)

Datavail is a leading provider of data management, application development, analytics, and cloud services, with more than 1,000 professionals helping clients build and manage applications and data via a world-class tech-enabled delivery platform and software solutions across all leading technologies. For more than 17 years, Datavail has worked with thousands of companies spanning different industries and sizes, and is an AWS Advanced Tier Consulting Partner, a Microsoft Solutions Partner for Data & AI and Digital & App Innovation (Azure), an Oracle Partner, and a MySQL Partner.

Datavail’s Data Management Services:

Datavail’s Data Management and Analytics practice is made up of experts who provide a variety of data services including initial consulting and development, designing and building complete data systems, as well as ongoing support and management of database, data warehouse, data lake, data integration, and virtualization and reporting environments. Datavail’s team is comprised of not just excellent BI & analytics consultants, but great people as well. Datavail’s data intelligence consultants are experienced, knowledgeable and certified in the best in breed BI and analytics software applications and technologies. We ascertain your business objectives, goals and requirements, assess your environment, and recommend the tools which best fit your unique situation. Our proven methodology can help your project succeed, regardless of stage. With the combination of a proven delivery model and top-notch experience ensures that Datavail will remain the Data Management experts on demand you desire. Datavail’s flexible and client focused services always add value to your organization.