Senior AI Product Manager

About Firmus

Firmus is the leading AI platform for preconstruction design review and risk analysis, helping construction teams identify design-related issues and mitigate financial risk. Recently acquired by The Nemetschek Group through its subsidiary Bluebeam, Firmus is uniquely positioned at the intersection of startup agility and enterprise scale. We now have access to massive resources, an established customer base, and deep industry expertise, while maintaining our fast-moving, innovative startup culture.

Key Responsibilities

Evaluation tooling (product ownership)

• Define and drive the roadmap for the AI agents, and tooling used to author and validate AI models
• Design and own the workflow from development to production.

Model quality & metrics

• Design the evaluation framework for our ML/AI systems, translating raw model outputs into unified business and product metrics.
• Perform statistical analysis and threshold calibration for optimization
• Define success criteria, KPIs, and acceptance thresholds and establish monitoring for models before and after production release

Cross-functional leadership

• Partner daily with SMEs to encode domain expertise into measurable checks
• Partner with AI engineers on evaluation methodology, dataset design, and error analysis
• Communicate model quality and risk to leadership and customers in business terms

Required Qualifications

• 5+ years product management, including 2+ years on ML/AI products
• Familiarity with Large Language Models (LLMs), machine learning concepts, and prompt engineering.
• Hands-on comfort with precision/recall, confusion matrices, threshold tuning, and statistical evaluation methods
• Strong statistical knowledge, experimental design, and problem-solving abilities
• Experience building internal tools or platforms (not just end-user features)
• Ability to translate between domain experts, engineers, and executives

To apply for this position, please submit your resume and cover letter to jobs@firmus.ai