Abstract
This UCF invention is a system and method that enables highly reliable and cost-effective inference for large language models deployed on distributed accelerator clusters. The approach safeguards the model’s runtime state (specifically the attention key value (KV) cache) by dynamically encoding it into redundant fragments distributed across multiple workers. When an accelerator fails, the system reconstructs the lost KV cache on a pre-initialized hot standby worker and seamlessly migrates the inference workload without restarting or recomputing prior tokens. By combining dynamic erasure coded KV cache protection with live migration, the invention transforms LLM inference from a fragile, failure prone process into a robust, high availability service suitable for mission critical commercial and enterprise applications.
Technical Details: The system enables fault-tolerant large language model inference across distributed accelerator clusters by protecting the attention key-value (KV) cache, which is essential for fast token generation. During normal operation, each active worker executes a model shard and maintains a local KV cache segment. These KV segments are optionally quantized, divided into stripes, and protected using dynamic erasure coding, producing redundant fragments that are distributed across other workers via high-speed interconnects.
A centralized controller continuously monitors worker health. When a failure is detected, the controller activates a hot-standby worker, gathers the encoded KV fragments from surviving workers, and reconstructs the failed worker’s KV cache on the standby. The corresponding model shard is then loaded, and inference requests are live-migrated to standby so generation continues without recomputing prior tokens. Encoding parameters and redundancy levels can be adjusted at runtime to balance performance, reliability, and resource usage
Benefit
High availability for LLM inference without costly restarts or replay of long prompts.Faster recovery than recomputation-based approaches, with bounded and predictable latency.Lower infrastructure and operational costs by improving accelerator utilization and avoiding full state replication.Improved energy efficiency and sustainability by reducing wasted computation and idle resources.Market Application
Cloud service providers offering high availability LLM inference and AI platform services.Enterprises and private AI deployments requiring reliable, always on inference for business-critical workflows.AI startups and SaaS companies seeking to reduce infrastructure costs while meeting strict uptime requirements.Edge and distributed AI systems, where failures are common and recovery time is constrained.
Brochure
