Principal Datadog Engineer

About This Role

This role focuses on establishing and scaling enterprise-wide observability using Datadog for a high-visibility federal program. The Principal Datadog Engineer will design, implement, and govern complex monitoring solutions, influencing platform architecture and guiding best practices.

Responsibilities

• Define and drive high-level observability strategies, addressing unique and complex challenges that impact system design and future innovations
• Architect and implement scalable Datadog solutions—dashboards, monitors, alerts, APM, synthetic testing, and log analysis—to enhance visibility across applications, infrastructure, and security layers
• Lead large-scale enterprise integrations and oversee the deployment of Datadog across multiple systems, ensuring consistency, reliability, and security
• Identify workflow inefficiencies and develop tools or methodologies to improve operational productivity, system reliability, and cost efficiency
• Serve as the escalation point for advanced Datadog performance and monitoring issues, performing root cause analysis with expertise in tracing, logs, and dashboards
• Mentor and develop engineering teams, promoting best practices and fostering continuous improvement through training and documentation
• Collaborate with cross-functional teams—including DevOps, SRE, Cloud, Security, and Application Development—to embed observability into all facets of operations
• Influence architecture decisions and lead technical governance efforts, ensuring alignment with program goals and industry standards
• Communicate complex technical concepts effectively to non-technical stakeholders, representing the organization’s technical capabilities externally when needed

Requirements

• Proven expertise in large-scale, enterprise-level software environments
• Deep, recent experience in Datadog architecture, configuration, and optimization
• Hands-on proficiency with Datadog dashboards, APM, synthetic tests, and integrations
• Proficiency in Python, Bash, PowerShell, YAML
• Experience building complex automation scripts and Lambda functions
• Demonstrated success in implementing infrastructure and application monitoring solutions using Datadog
• Cloud experience, preferably AWS
• Experience with Infrastructure-as-Code tools such as Terraform
• Prior experience as a Cloud DevOps or Automation Engineer with a track record of delivering scalable observability solutions
• Ability to obtain public trust clearance
• Legally authorized to work in the United States without employer sponsorship

Qualifications

• Bachelor’s Degree
• 10+ years of relevant experience, or 14+ years of relevant experience in lieu of a degree

Nice to Have

• Additional Datadog certifications
• Experience with DevSecOps practices and integrating security monitoring into observability platforms
• Background in modern, innovative engineering environments

Skills

* Required skills

Certifications