From a70a2ac3be652a49924deb09d24ccec257b7c236 Mon Sep 17 00:00:00 2001 From: oobabooga <112222186+oobabooga@users.noreply.github.com> Date: Sat, 24 Jun 2023 20:23:01 -0300 Subject: [PATCH] Update ExLlama.md --- docs/ExLlama.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/ExLlama.md b/docs/ExLlama.md index 1a51f188..6a0245b5 100644 --- a/docs/ExLlama.md +++ b/docs/ExLlama.md @@ -1,10 +1,18 @@ # ExLlama -## About +### About -ExLlama is an extremely optimized GPTQ backend ("loader") for LLaMA models. It features much lower VRAM usage and much higher speeds due to not relying on unoptimized transformers code. +ExLlama is an extremely optimized GPTQ backend for LLaMA models. It features much lower VRAM usage and much higher speeds due to not relying on unoptimized transformers code. -## Installation: +### Usage + +Configure text-generation-webui to use exllama via the UI or command line: + - In the "Model" tab, set "Loader" to "exllama" + - Specify `--loader exllama` on the command line + +### Manual setup + +No additional installation steps are necessary since an exllama package is already included in the requirements.txt. If this package fails to install for some reason, you can use the following manual procedure: 1) Clone the ExLlama repository into your `text-generation-webui/repositories` folder: @@ -14,8 +22,3 @@ cd repositories git clone https://github.com/turboderp/exllama ``` -2) Follow the remaining set up instructions in the official README: https://github.com/turboderp/exllama#exllama - -3) Configure text-generation-webui to use exllama via the UI or command line: - - In the "Model" tab, set "Loader" to "exllama" - - Specify `--loader exllama` on the command line