Senior AI Inference Engineer (LLM & Speech Models)

Senior AI Inference Engineer (LLM & Speech Models)

仕事概要

In shortYou'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.- **Team language:** English (Japanese not required)- **Salary:** ¥10M–¥20M per year, plus stock options- **Location:** Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.## You might be a fit if you are…- An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)- A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems- An engineer who has optimized throughput and cost for inference workloads on large GPU clusters## About the roleAt Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.This isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.On the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.## What this role is *not*- Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.- GPU cluster operations and general DevOps are handled by a separate team.- Writing CUDA kernels or building an inference engine from scratch is not the main job.- RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.## What you'll do### 1. Build a common inference framework for speech models- Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models- Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models- Build the inference APIs, SDKs and tooling our model teams use every day### 2. Optimize large-scale batch inference for audio- Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes- Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O- Measure and remove bottlenecks based on end-to-end processing time and cost### 3. Adapt and optimize LLM inference- Stand up LLM inference on mature engines such as vLLM and SGLang- Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache- Design scheduling that accounts for prefill/decode characteristics and task priority### 4. Plan GPU usage for inference workloads- Handle GPU allocation and capacity planning for batch inference and evaluation- Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray- Track and reduce GPU usage and inference cost per model and per taskWe're happy to go into more detail during the intro call and interviews.## Why this role- **Set the playbook in an area that doesn't have one yet.** Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.- **Own inference end to end.** You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.- **Directly move the business.** Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.## Tech stack- **Languages & frameworks:** Python, PyTorch- **LLM inference:** vLLM, SGLang, TensorRT-LLM, Triton Inference Server- **Speech models:** ASR, TTS, speech-to-speech, audio codecs, vocoders- **Orchestration:** Kubernetes, Slurm, Ray- **Low-level optimization (nice to have):** C++, CUDA, Triton, NCCL## Our productWe build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.Our edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.## Hiring processIntro call → Case assignment & first interview → Paid work trial* → Final interview → Offer meeting\*The work trial is 1–2 consecutive days at our Tokyo office, working on real tasks alongside the team, and it's paid. It's a chance for both sides to check fit and get a feel for the work.- **If you're based outside Japan,** we'll agree on the format together. We won't ask you to travel to Japan just for this step.- If your schedule or your current employer's side-work policy makes the trial difficult, just let us know.- We may also ask for references.## About RechoRecho develops foundation models for customer-support AI agents and also deploys them for clients, all within one organization. Our core strength is voice AI technology we built entirely in-house, which resolves the industry's long-standing trade-off between quality and cost.We've closed a Series A of approximately ¥700M and are rapidly rolling out to demanding production environments at major companies such as SBI Shinsei Bank and KDDI.## Learn more- [Corporate website](https://recho-ai.com/)- [Team interviews (note, in Japanese)](https://note.com/recho_ai)- [Company deck](https://recruit.recho-ai.com/)- [Case study: Partnership with KDDI and Altius Link (Japanese)](https://prtimes.jp/main/html/rd/p/000000021.000159800.html)- [Case study: SBI Shinsei Bank (Japanese)](https://mainichi.jp/articles/20260130/k00/00m/020/275000c)- [2-minute voice demo](https://recruit.recho-ai.com/#2f276fb799f480d5bba7d5784d6ad31e) — give it a listen!

必須スキル

In shortYou'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.- **Team language:** English (Japanese not required)- **Salary:** ¥10M–¥20M per year, plus stock options- **Location:** Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.## You might be a fit if you are…- An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)- A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems- An engineer who has optimized throughput and cost for inference workloads on large GPU clusters## About the roleAt Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.This isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.On the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.## What this role is *not*- Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.- GPU cluster operations and general DevOps are handled by a separate team.- Writing CUDA kernels or building an inference engine from scratch is not the main job.- RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.## What you'll do### 1. Build a common inference framework for speech models- Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models- Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models- Build the inference APIs, SDKs and tooling our model teams use every day### 2. Optimize large-scale batch inference for audio- Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes- Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O- Measure and remove bottlenecks based on end-to-end processing time and cost### 3. Adapt and optimize LLM inference- Stand up LLM inference on mature engines such as vLLM and SGLang- Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache- Design scheduling that accounts for prefill/decode characteristics and task priority### 4. Plan GPU usage for inference workloads- Handle GPU allocation and capacity planning for batch inference and evaluation- Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray- Track and reduce GPU usage and inference cost per model and per taskWe're happy to go into more detail during the intro call and interviews.## Why this role- **Set the playbook in an area that doesn't have one yet.** Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.- **Own inference end to end.** You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.- **Directly move the business.** Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.## Tech stack- **Languages & frameworks:** Python, PyTorch- **LLM inference:** vLLM, SGLang, TensorRT-LLM, Triton Inference Server- **Speech models:** ASR, TTS, speech-to-speech, audio codecs, vocoders- **Orchestration:** Kubernetes, Slurm, Ray- **Low-level optimization (nice to have):** C++, CUDA, Triton, NCCL## Our productWe build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.Our edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.## Hiring processIntro call → Case assignment & first interview → Paid work trial* → Final interview → Offer meeting\*The work trial is 1–2 consecutive days at our Tokyo office, working on real tasks alongside the team, and it's paid. It's a chance for both sides to check fit and get a feel for the work.- **If you're based outside Japan,** we'll agree on the format together. We won't ask you to travel to Japan just for this step.- If your schedule or your current employer's side-work policy makes the trial difficult, just let us know.- We may also ask for references.## About RechoRecho develops foundation models for customer-support AI agents and also deploys them for clients, all within one organization. Our core strength is voice AI technology we built entirely in-house, which resolves the industry's long-standing trade-off between quality and cost.We've closed a Series A of approximately ¥700M and are rapidly rolling out to demanding production environments at major companies such as SBI Shinsei Bank and KDDI.## Learn more- [Corporate website](https://recho-ai.com/)- [Team interviews (note, in Japanese)](https://note.com/recho_ai)- [Company deck](https://recruit.recho-ai.com/)- [Case study: Partnership with KDDI and Altius Link (Japanese)](https://prtimes.jp/main/html/rd/p/000000021.000159800.html)- [Case study: SBI Shinsei Bank (Japanese)](https://mainichi.jp/articles/20260130/k00/00m/020/275000c)- [2-minute voice demo](https://recruit.recho-ai.com/#2f276fb799f480d5bba7d5784d6ad31e) — give it a listen!

歓迎スキル

Nice-to-haves**Advanced models & speech**- Inference experience with audio generation models (diffusion, flow matching) or audio codecs- Hands-on experience across multiple model types, such as ASR, TTS and speech-to-speech- Interest or experience in building and optimizing RL rollout inference environments**Performance & low-level optimization**- Designing dynamic batching for variable-length audio, or distributed/parallel inference across multiple GPUs and nodes- Working knowledge of C++, CUDA, Triton and NCCL, and using profilers to pinpoint bottlenecks- Managing and optimizing GPU cost and compute budgets**Platform design & technical leadership**- Designing internal AI frameworks, inference libraries or model SDKs- Open-source contributions to vLLM, SGLang, PyTorch, Ray or similar- Setting technical roadmaps, design standards or code review guidelines**Languages**- Japanese or Chinese. Our team works in English, but Japanese (the company's working language) or Chinese (spoken by many of our colleagues) makes cross-team collaboration even smoother.

求める人物像

What we look for- **Hands-on:** You don't stop at architecture. You're on the front line writing the code and care about getting the implementation right.- **Data-driven:** You tackle performance and cost problems with profilers and benchmarks, not gut feel, and you optimize for the whole system.- **Pragmatic:** You don't reinvent the wheel. You pick the right open-source tools and frameworks and extend only what we actually need.- **Platform-minded:** You enjoy building tools and libraries that make other teams, especially model developers, more productive. That's Team-First in practice.## Our valuesThese are the values our team relies on to deliver Recho's value as fast as possible.- **Exceed Expectations:** We keep exceeding the expectations of customers and teammates by delivering both speed and quality.- **Team-First:** Each of us takes professional ownership, and we maximize the team's results, not individual ones.- **Continuous Feedback:** We engage seriously and speak candidly for the sake of results. The harder something is to say, the more we say it with honesty, respect and care.- **Earn Trust:** Trust is the foundation of every collaboration and every challenge. We act ethically and keep personal and company matters separate. Trust takes time to build and a moment to lose.

応募概要

給与
**Salary**Compensation is based on your current compensation, skills and experience.- Annual salary: ¥10M–¥20M- Salary review: Once a year, based on your contribution and company performance- Stock option plan

勤務地
WeWork Hanzomon (Hanzomon PREX North, 2-3-2 Kojimachi, Chiyoda-ku, Tokyo)Hybrid: mostly in the office, with some remote work. For candidates based outside Japan, remote arrangements are open to discussion.

雇用形態
Employment type: Full-time, permanent. Starting with a paid contract engagement first is also an option.

勤務体系
**Working hours**- Discretionary work system (professional type under Japanese labor law)- Deemed working hours: 9 hours/day- Start and end times are up to you. As a general guideline, the team's core working window is 10:00–19:00, adjusted as work requires.**Scope of duties and work location**This disclosure is required under Japanese labor law.- Duties: AI inference infrastructure development at hire; subject to change as determined by the company- Work location: Hanzomon WeWork at hire (some remote work possible); subject to change as determined by the company**Holidays & leave**Weekends, Japanese public holidays, year-end/New Year holidays (Dec 29–Jan 3), special leave, and paid annual leave (10 days in the first year)

試用期間
Probation period: 3 months, with no change in terms

福利厚生
**Insurance**- Full Japanese social insurance (health, pension, employment)- Commuting allowance (up to ¥25,000/month)**Office**- WeWork Hanzomon, 2 minutes from Hanzomon Station, with a dedicated floor, focus booths and meeting rooms- Free drinks, including coffee, soft drinks and alcohol- High-quality office chairs- MacBook Air and external monitor provided**Productivity**We build AI for our customers, so we work AI-native ourselves.- **AI tools support** - Engineering roles: Top-tier plans for Claude Code and Codex - Business & PM roles: Team plans that let you share context across your team- Amazon Business account for work-related books and equipment**Team & communication**Beyond project meetings and day-to-day conversations on Slack, we regularly set aside time for the whole company to share progress, discuss it together and get to know each other.- **Monday Kickoff:** Every Monday, we share everything from work updates to little things going on in our lives.- **Monthly All-Team Meeting:** On the first Friday of each month, each team shares its results from the previous month. Afterward, we get together over food.- **All Hands:** Twice a year, we share company direction and our mid- to long-term vision, bringing the whole organization together.**Smoking**- Indoor smoking prohibited; outdoor smoking area available