1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
|
# Local STT, TTS & Image Description Configuration Guide for LibreFang
This document describes how to configure and run the patched LibreFang package with local Speech-to-Text (STT), Text-to-Speech (TTS), and Image Description (vision) services.
---
## 1. Local Speech-to-Text (STT)
With the `feature-local-stt` patch, you can point Whisper-compatible transcription requests to a local service (like a custom python whisper-server or `llama.cpp` whisper server running on port `50090`).
### Configuration
Add the following to your `~/.librefang/config.toml` file:
```toml
[media]
audio_transcription = true
audio_provider = "openai" # Use OpenAI-compatible Whisper protocol
audio_model = "whisper-1" # Target model ID
audio_base_url = "http://localhost:50090/v1" # Local STT server base URL
```
And define `OPENAI_API_KEY` (even a dummy value like `"unused"`) in your `librefang.env` environment file:
```shell
OPENAI_API_KEY="unused"
```
---
## 2. Local Text-to-Speech (TTS)
With the `feature-local-tts` patch, you can point OpenAI-compatible text-to-speech requests to a local service (like Kokoro-FastAPI, F5-TTS, or other OpenAI-compatible TTS servers running on port `50095`).
### Configuration
Add the following to your `~/.librefang/config.toml` file:
```toml
[tts]
enabled = true
provider = "openai" # Use OpenAI-compatible protocol
[tts.openai]
model = "tts-1"
voice = "alloy"
format = "mp3"
speed = 1.0
base_url = "http://localhost:50095/v1" # Local TTS server base URL
```
Ensure standard credentials are set in `librefang.env` if the server requires them.
---
## 3. Local Image Description (Vision)
With the `feature-local-image` patch, you can point OpenAI-compatible vision/image-description requests to a local service (like `llama.cpp` llama-server with a LLaVA-style multimodal model running on port `50100`).
### Configuration
Add the following to your `~/.librefang/config.toml` file:
```toml
[media]
image_description = true
image_provider = "openai" # Use OpenAI-compatible Chat Completions protocol
image_model = "" # Target model ID (can be empty for local servers)
image_base_url = "http://localhost:50100/v1" # Local Vision server base URL
video_description = false
```
And define `OPENAI_API_KEY` (even a dummy value like `"unused"`) in your `librefang.env` environment file:
```shell
OPENAI_API_KEY="unused"
```
---
## 4. Local-Only `config.toml` Examples
Below are two configuration templates for a fully local-only inference pipeline (chat, embeddings, STT, TTS, and Vision).
### Scenario A: Single-Port Setup (Both LLM and Embeddings on the same port)
This configuration targets a unified local inference server (such as `llama-server` from `llama.cpp`) running on port `50080` that exposes both LLM and embeddings endpoints. This avoids using a separate placeholder provider (the "vllm hack").
```toml
# API Listen Address
api_listen = "127.0.0.1:4545"
api_key = "my-secure-api-key"
# 1. Chat (LLM) configuration targeting local unified server
[default_model]
provider = "openai"
model = "qwen3"
api_key_env = "UNUSED_API_KEY"
base_url = "http://localhost:50080/v1"
# 2. Vector Memory & Embeddings targeting the same unified server
[memory]
embedding_provider = "openai"
embedding_model = "qwen3-embedding"
embedding_dimensions = 1536
# 3. Speech-to-Text & Image Description (e.g. whisper-server on 50090, vision-server on 50100)
[media]
audio_transcription = true
audio_provider = "openai"
audio_model = "whisper-1"
audio_base_url = "http://localhost:50090/v1"
image_description = true
image_provider = "openai"
image_base_url = "http://localhost:50100/v1"
image_model = ""
video_description = false
# 4. Text-to-Speech (e.g. kokoro-tts-server on port 50095)
[tts]
enabled = true
provider = "openai"
[tts.openai]
model = "tts-1"
voice = "alloy"
base_url = "http://localhost:50095/v1"
# 5. Base URL Overrides mapping all "openai" requests to the local port
[provider_urls]
openai = "http://localhost:50080/v1"
```
### Scenario B: Multi-Port Setup with vLLM (vLLM for Chat, llama-server for Embeddings)
This configuration applies when transitioning the chat model to a local `vLLM` server running on port `50080`, while keeping vector embeddings separated on `llama-server` (or another embedding server) on port `50085`.
```toml
# API Listen Address
api_listen = "127.0.0.1:4545"
api_key = "my-secure-api-key"
# 1. Chat (LLM) configuration targeting vLLM on port 50080
[default_model]
provider = "vllm"
model = "qwen3"
api_key_env = "UNUSED_API_KEY"
base_url = "http://localhost:50080/v1"
# 2. Vector Memory & Embeddings targeting llama-server on port 50085
[memory]
embedding_provider = "openai"
embedding_model = "qwen3-embedding"
embedding_dimensions = 1536
# 3. Speech-to-Text & Image Description (e.g. whisper-server on 50090, vision-server on 50100)
[media]
audio_transcription = true
audio_provider = "openai"
audio_model = "whisper-1"
audio_base_url = "http://localhost:50090/v1"
image_description = true
image_provider = "openai"
image_base_url = "http://localhost:50100/v1"
image_model = ""
video_description = false
# 4. Text-to-Speech (e.g. kokoro-tts-server on port 50095)
[tts]
enabled = true
provider = "openai"
[tts.openai]
model = "tts-1"
voice = "alloy"
base_url = "http://localhost:50095/v1"
# 5. Base URL Overrides mapping (targets the embedding port 50085 for "openai")
[provider_urls]
openai = "http://localhost:50085/v1"
```
|