LOADING

Building a self-hosted LLM ecosystem in Rust: from infrastructure to applications

This talk explores building a complete self-hosted LLM stack in Rust: Paddler, a distributed load balancer for serving LLMs at scale, and Poet, a static site generator that consumes those LLMs for AI-powered content features.

Mateusz Charytoniuk
Software Architect
About This Talk

This talk explores building a complete self-hosted LLM stack in Rust: Paddler, a distributed load balancer for serving LLMs at scale, and Poet, a static site generator that consumes those LLMs for AI-powered content features.

We'll dive into the hard problems: async request routing across dynamic agent fleets, integrating with llama.cpp's C++ codebase, managing KV cache in custom slots, and implementing zero-to-N autoscaling with request buffering. You'll see how Rust's ownership model prevented entire classes of bugs in distributed state management, and walk away with concrete patterns for building and consuming LLM infrastructure in production.

more great talks

Might Be Interesting

Day 1
  —  
8:00 pm
arrow pointing right icon

Rustikon afterparty

Join us at the Rustikon afterparty!

Day 2
  —  
11:50 am
arrow pointing right icon

Let's build an async runtime together

During this talk we'll build a basic, working async runtime using nothing more than a standard library. The point? To see it's approachable for mere mortals.

Day 2
  —  
11:15 am
arrow pointing right icon

Rust performance debugging with TUIs and LLMs

In my session, I will present the https://hotpath.rs crate and explain how it compares to other profiling tools available.

See All Events
Join us!

We're looking for amazing speakers.
CFP is open till 10.01.2023

Fill in Call for Papers
location icon

Location

Centrum Konferencyjne POLIN, Poland
stay in touch icon

Follow Us

Contact Us