Enterprise Latency Optimization, Streaming Infrastructure, and Retrieval-Augmented Generation Architectures: Research Innovations in Enterprise Conversational AI Performance
Keywords:
latency optimization; retrieval-augmented generation; streaming infrastructure; semantic caching; percentile benchmarking; pipeline orchestrationAbstract
Enterprise conversational AI platforms now serve millions of concurrent users, and latency has become a structural design constraint rather than a post-launch tuning target. Most published optimization research addresses a narrow slice of this problem: how quickly a single language model call completes, through techniques such as continuous batching, paged memory management, and speculative decoding. This paper argues that orchestration and routing overhead, not model inference itself, is the dominant addressable latency bottleneck in production enterprise conversational AI, and that the field's heavy concentration on model-level techniques is somewhat misallocated relative to the parallelism available at the pipeline level. We introduce a pipeline decomposition formula, TM99 = WebSocket + max(LLM Routing, Contextualization + max(Cache Lookup, Guardrail)) + Plugin Execution + Response Synthesis, that expresses end-to-end tail latency as a function of sequential and parallelizable stages, and we describe the three-path execution model used to realize that parallelism in production. Drawing on the caching, routing, and speculative-execution literature, we survey the measured gains reported there, ranging from single-digit-percentage improvements to more than sixty-fold reductions in time to first token depending on technique and cache hit rate, and we argue for TM50/TM90/TM99 percentile benchmarking over mean-latency reporting, which obscures the tail behavior that governs service-level compliance. The analysis indicates that orchestration-layer parallelism frequently exceeds available model-level gains in enterprise multi-agent settings, and we outline where practitioners should concentrate optimization effort accordingly.
Downloads
References
A. Vaswani et al., "Attention is all you need," in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 5998-6008.
P. Lewis et al., "Retrieval-augmented generation for knowledge-intensive NLP tasks," in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 9459-9474.
R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, "Efficiently scaling transformer inference," Proceedings of Machine Learning and Systems, vol. 5, 2023.
G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, "Orca: A distributed serving system for transformer-based generative models," in Proc. 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022, pp. 521-538.
W. Kwon et al., "Efficient memory management for large language model serving with PagedAttention," in Proc. 29th ACM Symposium on Operating Systems Principles (SOSP), 2023.
Y. Leviathan, M. Kalman, and Y. Matias, "Fast inference from transformers via speculative decoding," in Proc. 40th International Conference on Machine Learning (ICML), 2023.
C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, "Accelerating large language model decoding with speculative sampling," arXiv preprint arXiv:2302.01318, 2023.
F. Bang, "GPTCache: An open-source semantic cache for LLM applications," in Proc. 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS), 2023, pp. 212-218.
V. J. Reddi et al., "MLPerf inference benchmark," in Proc. ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020, pp. 446-459.
X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, "Towards efficient generative large language model serving: A survey from algorithms to systems," arXiv preprint arXiv:2312.15234, 2023.
Y. Wang, K. Chen, H. Tan, and K. Guo, "Tabi: An efficient multi-level inference system for large language models," in Proc. 18th European Conference on Computer Systems (EuroSys), 2023, pp. 233-248.
G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, "Efficient streaming language models with attention sinks," in Proc. International Conference on Learning Representations (ICLR), 2024.
Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, "DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving," in Proc. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.
P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini, "Splitwise: Efficient generative LLM inference using phase splitting," in Proc. ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118-132.
A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, "Self-RAG: Learning to retrieve, generate, and critique through self-reflection," in Proc. International Conference on Learning Representations (ICLR), 2024.
C. Jin, Z. Zhang, X. Jiang, F. Liu, X. Liu, X. Liu, and X. Jin, "RAGCache: Efficient knowledge caching for retrieval-augmented generation," arXiv preprint arXiv:2404.12457, 2024.
I. Gim, G. Chen, S. Lee, N. Sarda, A. Khandelwal, and L. Zhong, "Prompt cache: Modular attention reuse for low-latency inference," Proceedings of Machine Learning and Systems, vol. 6, 2024.
I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica, "RouteLLM: Learning to route LLMs with preference data," in Proc. International Conference on Learning Representations (ICLR), 2025.
D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruhle, L. V. S. Lakshmanan, and A. H. Awadallah, "Hybrid LLM: Cost-efficient and quality-aware query routing," in Proc. International Conference on Learning Representations (ICLR), 2024.
L. Chen, M. Zaharia, and J. Zou, "FrugalGPT: How to use large language models while reducing cost and improving performance," arXiv preprint arXiv:2305.05176, 2023.
J. Chen, H. Lin, X. Han, and L. Sun, "Benchmarking large language models in retrieval-augmented generation," in Proc. AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024.
X. Yang et al., "CRAG: Comprehensive RAG benchmark," in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
