LocalAI

Links

• LocalAI homepage
• LocalAI FAQ
• Check out Open webui

HOWTO

List models

curl http://localhost:8080/v1/models

Audio to text

• https://localai.io/features/audio-to-text/
• https://docs.llamaindex.ai/en/stable/examples/llm/localai/

Apply model

curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{
  "id": "huggingface@TheBloke/Yarn-Mistral-7B-128k-GGUF/yarn-mistral-7b-128k.Q5_K_M.gguf"
}'

Scripts

Talk to the chat interface

#!/bin/bash
echo -n "Ask me anything: "
read A
curl -s http://localhost:8080/v1/chat/completions \
   -H "Content-Type: application/json" \
   -d '{ "model": "gpt-4", "messages": [{"role": "user", "content": "'"$A"'", "temperature": 0.1}] }' |\
    jq     '.choices[].message.content' | sed 's/\\n/\n/g' | sed 's/\\"/"/g'

FAQ

File/directory locations

Local-ai log

 /usr/share/local-ai/llama.log

Messages

GPU device found but no CUDA backend present

If running in docker, try restarting docker