Senior Software Engineer II Applied AI and Evaluations
Smartsheet
πΏπ¦ Hirezar Summary for South African Applicants
This fully remote full time position at Smartsheet is open to applicants from South Africa. As a remote position, you can work from anywhere in South Africa β whether you're based in Johannesburg, Cape Town, Durban, or a smaller town.
Job Description
For over 20 years, Smartsheet has helped people and teams achieve-well, anything. From seamless work management to smart, scalable solutions, we've always worked with flow. We're building tools that empower teams to automate the manual, uncover insights, and scale smarter. But more than that, we're creating space- space to think big, take action, and unlock the kind of work that truly matters. Because when challenge meets purpose, and passion turns into progress, that's magic at work, and it's what we show up for everyday.
Smartsheet is building the next generation of AI-powered work management through SmartAssist, our intelligent agent platform. As we scale from early demos to production-grade agents, quality is the critical frontier and we are looking for a Agent Quality Engineer to own it.
This is not a QA role. It's a deeply technical, high-autonomy position at the intersection of LLM evaluation, prompt and context engineering, and retrieval-augmented generation. You will diagnose why our agents fail, design the systems that catch regressions, and drive measurable improvements across our orchestrator and subagent fleet.
You will work closely with our Agent Engineering and AI Platform teams, embedded in a team that has already shipped evaluation infrastructure on Databricks/MLflow and is building toward a mature Agent Development Lifecycle (ADLC).
You Will:
- Own agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist's orchestrator and subagents
- Identify failure modes across quality dimensions factual accuracy, completeness, tone, actionability, and latency and prioritize what to fix
- Drive quality improvements through prompt engineering,
Please mention the word **FRESHER** and tag RMjYwNDoyZGMwOjEwMToyMDA6OjI4NTA= when applying to show you read the job post completely (#RMjYwNDoyZGMwOjEwMToyMDA6OjI4NTA=). This is a beta feature to avoid spam applicants. Companies can search these words to find applicants that read this and see they're human.
Tips for South African Applicants
Timezone Advantage
South Africa (SAST, UTC+2) overlaps well with European business hours and has a few hours of overlap with US East Coast. Mention your timezone flexibility in your application.
Salary in Context
Even without a listed salary, international remote roles typically pay 2-3x more than equivalent local positions in South Africa due to the exchange rate advantage.
Application Tips
Tailor your CV to international standards β use a clean format, highlight remote work experience, and include your English proficiency. Many SA applicants succeed by emphasising their strong work ethic and cultural adaptability.
Load Shedding Preparedness
If you're applying for a remote role, having a backup power solution (UPS, inverter, or generator) and mobile data as a backup internet connection shows employers you're prepared for South Africa's infrastructure challenges.
About Smartsheet
Smartsheet is a company in the director industry that hires remote workers from South Africa. They currently have 3 open positions on Hirezar. View all Smartsheet jobs β