LOADING

Building a self-hosted LLM ecosystem in Rust: from infrastructure to applications

This talk explores building a complete self-hosted LLM stack in Rust: Paddler, a distributed load balancer for serving LLMs at scale, and Poet, a static site generator that consumes those LLMs for AI-powered content features.

Mateusz Charytoniuk
Software Architect
About This Talk

This talk explores building a complete self-hosted LLM stack in Rust: Paddler, a distributed load balancer for serving LLMs at scale, and Poet, a static site generator that consumes those LLMs for AI-powered content features.

We'll dive into the hard problems: async request routing across dynamic agent fleets, integrating with llama.cpp's C++ codebase, managing KV cache in custom slots, and implementing zero-to-N autoscaling with request buffering. You'll see how Rust's ownership model prevented entire classes of bugs in distributed state management, and walk away with concrete patterns for building and consuming LLM infrastructure in production.

more great talks

Might Be Interesting

Day 2
  —  
12:25 pm
arrow pointing right icon

Clean Code for Good Science: Rust in Research and Health

This talk explores what it means to write scientific software that lives up to the standards we expect of science itself.

Day 2
  —  
2:00 pm
arrow pointing right icon

Getting hired in Rust - Recruiter's perspective

I’ll share what the Rust job market really looks like in 2025 — where companies are hiring, which skills stand out, and how the recruitment process actually works behind the scenes.

Day 2
  —  
3:10 pm
arrow pointing right icon

No cars were harmed for this talk: Automotive Rust, amateur style

I'll initiate you in the art of 'CAN bus sniffing': Connecting to the central nervous system of a modern car, interpreting the data, and seeing what we can build as enthousiastic amateurs.

Day 1
  —  
11:15 am
arrow pointing right icon

Supercharging Axum Development with Claude Code: Practical AI-Assisted Workflows for Rust Engineers

In this talk, we’ll explore battle-tested best practices for integrating Claude Code into a professional Axum development workflow without compromising on Rust’s core values: correctness, clarity, and maintainability.

See All Events
Join us!

We're looking for amazing speakers.
CFP is open till 10.01.2023

Fill in Call for Papers
location icon

Location

Centrum Konferencyjne POLIN, Poland
stay in touch icon

Follow Us

Contact Us