Posted:
Job Description
Senior DevOps with Infrastructure
Role: Senior DevOps with Infrastructure
Type: Remote - working EST Hours
Security Clearance: Must be eligible for up to a Top-Secret Security Clearance
Job Overview
The Senior DevOps Engineer is a critical hands-on role responsible for the architecture, deployment, and performance optimization of a cloud-native, real-time data processing platform.
Key Responsibilities
- Design, deploy, and manage the underlying data platform using Kubernetes to ensure high-scalability and resiliency for distributed data workload.
- Build and maintain high-throughput ETL/ELT pipelines using Python and Apache Spark.
- Implement and tune the distributed SQL query engine Trino against the Iceberg table format to optimize query performance and data reliability.
- Develop core services, APIs, and automation tooling for the data platform primarily using the C# programming language, ensuring speed and reliability.
- Architect and implement robust authentication and authorization integrations across the platform, including seamless connectivity via LDAP/SSO protocols.
- Utilize Infrastructure-as-Code (e.g., Terraform, Helm) and GitOps principles to manage the platform lifecycle and enforce immutable infrastructure practices.
- Work closely with security teams to implement and audit IAM policies and ensure container security best practices are followed.
Must-Have:
- 7+ years of experience in DevOps in Infrastructure with hands-on experience with Kubernetes administration, tuning, and deployment in production environments.
- C# for building platform services and integration points.
- Deploying, configuring, and optimizing Trino for large, federated data sets.
- Implementing Iceberg table format for data lake transactions and schema evolution.
- Python and mandatory practical experience with Apache Spark for data processing.
- Integrating enterprise authentication and authorization protocols (LDAP/SSO) into microservices.