Inference Awesome
约 1 分钟阅读
标签：资源精选
- ollama
- toruchrun
- vLLM Virtual Large Language Model
- PagedAttention
- SGLang
- llama.cpp
- trymirai/uzuGitHubtrymirai/uzu
- localai
- NVIDIA/TensorRT-LLMGitHubNVIDIA/TensorRT-LLM
- Apache-2.0, C++, Python
- trt
- huggingface/text-generation-inferenceGitHubhuggingface/text-generation-inferencehuggingface/text-generation-inference · 支持硬件 · Nvidia · AMD (-rocm) · Inferentia · Intel GPU · Gaudi · Google TPU笔记：Text Generation Inference
- Apache-2.0, Python, Rust
- HF TGI
- triton-inference-server/serverGitHubtriton-inference-server/server
- BSD-3, Python, C++
- NVIDIA Triton
- bentoml/BentoMLGitHubbentoml/BentoML
- Apache-2.0, Python
- Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines
- bentoml/BentoDiffusionGitHubbentoml/BentoDiffusion
- bentoml/OpenLLMGitHubbentoml/OpenLLM
- Image
- ComfyUI
- AUTOMATIC1111/stable-diffusion-webuiGitHubAUTOMATIC1111/stable-diffusion-webuiAUTOMATIC1111/stable-diffusion-webui · Mikubill/sd-webui-controlnet · sontungdo/sd-image-editor · 图片编辑，截取一部分来做生成 · modelscope/facechain笔记：stable-diffusion-webui
- A1111
- SD
- Audio
- Whisper
- Embeddings
- michaelfeil/infinityGitHubmichaelfeil/infinity
- MIT, Python
- michaelfeil/infinityGitHubmichaelfeil/infinity
- exo-explore/exoGitHubexo-explore/exo
- GPLv3, Python
- Tencent/ncnnGitHubTencent/ncnn
- BSD-3, C/C++
- neural network inference framework optimized for the mobile platform
- InternLM/lmdeployGitHubInternLM/lmdeploy
- Apache-2.0, Python, C++
- based on MMRazorGitHubopen-mmlab/mmrazor and MMDeployGitHubopen-mmlab/mmdeploy
- mit-han-lab/nunchakuGitHubmit-han-lab/nunchaku
- Apache-2.0, Python, C++
- Nunchaku is a high-performance inference engine optimized for 4-bit neural networks
- SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
- theroyallab/tabbyAPIGitHubtheroyallab/tabbyAPI
- AGPLv3, Python
- turboderp-org/exllamav2GitHubturboderp-org/exllamav2
- MIT, Python
- turboderp-org/exllamav3GitHubturboderp-org/exllamav3
- MIT, Python
- Reading
关联信息
反向链接、本文链接的其他页面和外部资料。
反向链接
- 推理笔记 · Inference Awesome
Inference Awesome · 模型大小与架构 (Model Size & Architecture) · 参数量 · 模型的层数、宽度等 · 精度与量化 (Precision & Quantization) · 原始精度 - FP32, BF16
参考资料
GitHub19 条
- AUTOMATIC1111/stable-diffusion-webuigithub.com/AUTOMATIC1111/stable-diffusion-webui
- bentoml/BentoDiffusiongithub.com/bentoml/BentoDiffusion
- bentoml/BentoMLgithub.com/bentoml/BentoML
- bentoml/llm-inference-handbookgithub.com/bentoml/llm-inference-handbook
- bentoml/OpenLLMgithub.com/bentoml/OpenLLM
- exo-explore/exogithub.com/exo-explore/exo
- huggingface/text-generation-inferencegithub.com/huggingface/text-generation-inference
- InternLM/lmdeploygithub.com/InternLM/lmdeploy
- michaelfeil/infinitygithub.com/michaelfeil/infinity
- mit-han-lab/nunchakugithub.com/mit-han-lab/nunchaku
- NVIDIA/TensorRT-LLMgithub.com/NVIDIA/TensorRT-LLM
- open-mmlab/mmdeploygithub.com/open-mmlab/mmdeploy
- open-mmlab/mmrazorgithub.com/open-mmlab/mmrazor
- Tencent/ncnngithub.com/Tencent/ncnn
- theroyallab/tabbyAPIgithub.com/theroyallab/tabbyAPI
- triton-inference-server/servergithub.com/triton-inference-server/server
- trymirai/uzugithub.com/trymirai/uzu
- turboderp-org/exllamav2github.com/turboderp-org/exllamav2
- turboderp-org/exllamav3github.com/turboderp-org/exllamav3