harleyszhang/rapid_llm

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

语言：Python · GitHub 星标：195 · Fork：33

话题：attention · continues · continuous-batching · flash-attention-3 · llama3 · llm · llm-inference · python3

在 GitHub 上查看原仓库