Koko210/miku-discord
1
0
Fork 0
Code Issues 34 Pull Requests Actions Packages Projects Releases Wiki Activity
120 Commits 1 Branch 0 Tags
d5b9964ce7e2148a253cea17e91b2c8751993c69
Commit Graph

4 Commits

Author SHA1 Message Date
koko210Serve
eafab336b4 feat: add Traefik proxy, custom chat template, improve Cheshire Cat memory 
docker-compose.yml:
- Add Traefik proxy network + labels for miku.panel domain
- Connect miku-bot service to proxy network

llama-swap-config.yaml / llama-swap-rocm-config.yaml:
- Add --chat-template-file flag to disable Llama 3.1 built-in tool
  calling (was causing malformed responses)
- ROCm config: add Rocinante-X 12B model entry for comparison testing

cheshire-cat discord_bridge plugin:
- Increase declarative memory recall (k=3→10, threshold 0.7→0.5)
  for better factual retrieval
- Add agent_prompt_prefix hook to enforce factual accuracy from
  declarative memories
- Add before_agent_starts debug logging for memory inspection
- Add passthrough hooks for message/suffix pipeline
 2026-03-04 00:48:58 +02:00
koko210Serve
c0aaab0c3a Disabled KV cache offloading on llama-server and enabled Flash Attention. Performance gains in the tens. 2026-01-27 19:11:49 +02:00
koko210Serve
fe0962118b Implemented new Japanese only text mode with WebUI toggle, utilizing a llama3.1 swallow dataset model. Next up is Japanese TTS. 2026-01-23 15:02:36 +02:00
koko210Serve
1fc3d74a5b Add dual GPU support with web UI selector 
Features:
- Built custom ROCm container for AMD RX 6800 GPU
- Added GPU selection toggle in web UI (NVIDIA/AMD)
- Unified model names across both GPUs for seamless switching
- Vision model always uses NVIDIA GPU (optimal performance)
- Text models (llama3.1, darkidol) can use either GPU
- Added /gpu-status and /gpu-select API endpoints
- Implemented GPU state persistence in memory/gpu_state.json

Technical details:
- Multi-stage Dockerfile.llamaswap-rocm with ROCm 6.2.4
- llama.cpp compiled with GGML_HIP=ON for gfx1030 (RX 6800)
- Proper GPU permissions without root (groups 187/989)
- AMD container on port 8091, NVIDIA on port 8090
- Updated bot/utils/llm.py with get_current_gpu_url() and get_vision_gpu_url()
- Modified bot/utils/image_handling.py to always use NVIDIA for vision
- Enhanced web UI with GPU selector button (blue=NVIDIA, red=AMD)

Files modified:
- docker-compose.yml (added llama-swap-amd service)
- bot/globals.py (added LLAMA_AMD_URL)
- bot/api.py (added GPU selection endpoints and helper function)
- bot/utils/llm.py (GPU routing for text models)
- bot/utils/image_handling.py (GPU routing for vision models)
- bot/static/index.html (GPU selector UI)
- llama-swap-rocm-config.yaml (unified model names)

New files:
- Dockerfile.llamaswap-rocm
- bot/memory/gpu_state.json
- bot/utils/gpu_router.py (load balancing utility)
- setup-dual-gpu.sh (setup verification script)
- DUAL_GPU_*.md (documentation files)
 2026-01-09 00:03:59 +02:00