JobCubby JobCubby
Sign up
Red Cedar Consultancy

Senior Data Engineer

Red Cedar Consultancy District of Columbia, United States

IT Services and IT Consulting · 51-200 employees

Oct 09
data-engineer Mid (2-5 yrs) Full-time United States
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.

About the role

The Senior Data Engineer will design and optimize high-performance Spark applications and enterprise data infrastructure within AWS Databricks. They are responsible for managing ETL/ELT pipelines, administering Databricks workspaces, and ensuring security compliance in a federal environment.

What they look for

Databricks PySpark Python SQL AWS Delta Lake ETL/ELT Terraform CI/CD Medallion Architecture Unity Catalog Data Engineering Agile Scrum FISMA FedRAMP

Requirements

Candidates must have at least 3 years of hands-on Databricks experience and proficiency in Python, PySpark, and SQL. A bachelor's degree in a relevant engineering field and an active High-Risk Public Trust (Tier 4) clearance are required.

Full description

Job Title: Senior Data Engineer

Location: Washinton DC

Summary

The Senior Data Engineer / Databricks Specialist will support a federal project by developing, designing , and optimizing high-performance Spark applications and enterprise data infrastructure in AWS Databricks. The role focuses on scalable ETL/ELT pipelines using Python, PySpark, SQL, and Delta Lake; Medallion Architecture; and secure, reliable data solutions for mission-critical environments.

Responsibilities

  • Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
  • Design, build, and maintain ETL/ELT pipelines using Medallion Architecture (Bronze, Silver, and Gold layers), ensuring data integrity, completeness, and lineage.
  • Administer Databricks workspaces, clusters, scheduled jobs, access controls, Unity Catalog, and notebooks.
  • Manage AWS services including S3, Lambda, IAM, Aurora, and Glue; provision infrastructure with Terraform.
  • Optimize queries and Spark jobs; troubleshoot and debug large-scale data processing solutions.
  • Integrate structured, semi-structured, and unstructured data from APIs and databases.
  • Integrate security and compliance controls into DevSecOps CI/CD pipelines using Jenkins, GitLab CI, or equivalent tools, supporting FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
  • Implement data quality checks, anomaly detection, validation rules, and automated lineage tracking.
  • Collaborate with cross-functional Agile/Scrum teams in multi-contractor federal environments; provide technical leadership and production on-call support.

Required Qualifications

  • At least 3 years of dedicated, hands-on Databricks development experience.
  • Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
  • Strong proficiency in Python and SQL and working knowledge of REST API-based data ingestion.
  • Strong hands-on experience with AWS S3, IAM, Lambda, Glue, and Aurora.
  • Production experience with Terraform infrastructure as code and CI/CD tools such as Jenkins, GitLab CI, or equivalent.
  • Experience working in Agile/Scrum delivery environments.
  • Bachelor's degree in Computer Science, Information Technology, Data Science, or a related engineering discipline.
  • Active or previous High-Risk Public Trust (Tier 4) clearance.

Preferred Qualifications

  • Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and MLflow or Model Registry for AI/ML pipelines.
  • Experience supporting federal government contracts and FISMA/FedRAMP environments.

Preferred Certifications

  • Databricks Certified Data Engineer Associate
  • Databricks Certified Data Engineer Professional
  • Databricks Certified Machine Learning Associate
  • AWS Certified Solutions Architect – Associate or Professional
  • AWS Certified Data Analytics – Specialty

Similar roles