Principal Engineer

The Principal Engineer is a senior technical leader responsible for driving the design, implementation, and management of scalable, secure, and highly available AI solutions that power AOT s portfolio of digital products for the public and private sectors. This role blends deep technical expertise with strategic thinking and hands-on leadership setting architectural direction, championing AI-assisted engineering practices, and mentoring engineers to build the next generation of intelligent, cloud-native applications. The Principal Engineer partners closely with the Head of Engineering, product leaders and cross-functional teams to translate complex business and user needs into resilient, future-ready technical solutions grounded in open-source principles and modern cloud architecture. Job Duties and Responsibilities Lead the end-to-end design, architecture and delivery of scalable, secure and highly available AI-powered solutions across AOT s product portfolio. Define and evolve the technical vision for AI/ML platforms, including LLM-based applications, RAG pipelines, agentic workflows and ML model lifecycle management. Architect distributed, cloud-native systems leveraging AWS, Azure, and/or GCP ensuring high availability, fault tolerance, observability and cost efficiency across multi-cloud environments. Champion the adoption of AI-assisted development practices (e.g.Claude Code, Cursor, GitHub Copilot or other agentic coding tools) to accelerate engineering velocity and elevate code quality across teams. Drive the evaluation, selection and integration of open-source frameworks, libraries, and platforms to build sustainable, vendor-agnostic solutions aligned with AOT s hybrid-source philosophy. Provide hands-on technical leadership writing production-grade code, building proof-of-concepts and prototypes, reviewing pull requests and shaping critical design decisions across multiple teams. Establish engineering excellence standards across security, performance, scalability, observability, accessibility and privacy compliance with special focus on responsible and ethical AI practices. Mentor and grow senior engineers, tech leads and architects through coaching, design reviews, technical deep-dives and structured feedback building a strong engineering bench. Partner with the Head of Engineering, Product and Solutions teams to align technical strategy with business outcomes, customer needs and long-term product roadmaps. Lead cross-team architectural reviews and serve as the principal authority on technical trade-offs, build-vs-buy decisions and platform standardization. Define and guide DevOps, SRE and MLOps practices including CI/CD, infrastructure- as-code, model deployment, monitoring and continuous evaluation of AI systems. Stay at the forefront of emerging trends in Generative AI, LLMs, agentic systems, and applied ML actively bringing innovation back into AOT s products and engineering culture. Represent AOT s engineering organization in technical forums, customer conversations, open-source communities, and industry events as needed. Champion a culture of psychological safety, continuous learning, experimentation, open knowledge-sharing, and shared ownership of outcomes. Required Qualifications Education Bachelor s or Master s degree in Computer Science, Engineering, or a related field, or equivalent professional experience. Experience 12 years of professional software engineering experience, with at least 3 years in a Principal Engineer, Staff Engineer or Enterprise Architect role. Proven track record of architecting and delivering large-scale, production-grade AI/ML or data-intensive solutions in cloud environments. Demonstrated experience designing and operating highly available, distributed systems serving enterprise or public sector customers. Hands-on experience leading the productionization of AI/ML workloads including LLM-based applications, RAG systems, vector databases, prompt engineering, fine- tuning or agentic workflows. Technical Skills Deep expertise in scalable architecture patterns microservices, event-driven systems, distributed computing, caching, queueing and high-throughput data pipelines. Strong proficiency across multiple cloud platforms (AWS, Azure, GCP) including compute, storage, networking, identity, container orchestration (Kubernetes, ECS, AKS), and managed AI/ML services (Bedrock, SageMaker, Azure OpenAI, Vertex AI). Strong hands-on coding ability in one or more modern backend languages/frameworks such as Python, Java, Node.js or Go. Working knowledge of frontend frameworks such as React, Vue, or Angular, and the ability to reason about full-stack performance and architecture. Practical experience with AI/ML frameworks and ecosystems PyTorch, TensorFlow, scikit-learn, Hugging Face, LangChain/LlamaIndex, OpenAI/Anthropic/Bedrock APIs, vector databases. Strong command of AI-assisted development tools and workflows (Claude Code, GitHub Copilot, Cursor or any other agentic coding agents) and ability to operationalize them across an engineering organization. Leadership Communication Proven ability to lead through influence driving alignment across engineering teams, product, design and executive stakeholders without direct reporting authority. Excellent written and verbal communication skills, with the ability to translate complex technical concepts for both technical and non-technical audiences. Strong mentorship and coaching skills, with a track record of growing senior engineers and tech leads. Solid understanding of DevOps, SRE and MLOps practices CI/CD, IaC (Terraform, CloudFormation), containerization (Docker, Kubernetes), observability (Prometheus, Grafana, OpenTelemetry) and model monitoring. Deep understanding of secure coding practices, identity and access management, data privacy, accessibility and compliance frameworks relevant to enterprise and public sector software. Experience with process orchestration and workflow tools (e.g., M8Flow, Camunda, Airflow, Temporal) is a strong plus. Skilled in agile development methodologies and iterative, customer-centric delivery.