Package Details: koboldcpp-cuda 1.53-1

Git Clone URL: https://aur.archlinux.org/koboldcpp-cuda.git (read-only, click to copy)
Package Base: koboldcpp-cuda
Description: Full-featured CUDA build of koboldcpp
Upstream URL: https://github.com/LostRuins/koboldcpp
Keywords: ai llm,
Licenses: AGPL3
Submitter: alpindale
Maintainer: alpindale
Last Packager: alpindale
Votes: 2
Popularity: 0.76
First Submitted: 2023-10-23 17:51 (UTC)
Last Updated: 2023-12-25 21:21 (UTC)

Latest Comments

ilikenwf commented on 2024-02-23 04:13 (UTC)

# Maintainer: Alpin <alpin 'at' alpindale 'dot' dev>
# Author: LostRuins (concedo)
pkgname=koboldcpp-cuda
pkgver=1.58
pkgrel=1
pkgdesc="Full-featured CUDA build of koboldcpp"
arch=('x86_64')
url="https://github.com/LostRuins/koboldcpp"
license=('AGPL3')
depends=('python' 'cblas' 'openblas' 'clblast' 'cuda' 'tk' 'customtkinter')
optdepends=('customtkinter')
source=("$pkgname-$pkgver.tar.gz::https://github.com/LostRuins/koboldcpp/archive/refs/tags/v$pkgver.tar.gz")
sha256sums=('445df325d5d3d6874758c32e5eb88a9d2b0a75451ff9467b45f9c38d766fb828')

build() {
  cd "$srcdir/koboldcpp-$pkgver"
  make LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1 LLAMA_CUBLAS=1
}

package() {
  cd "$srcdir/koboldcpp-$pkgver"

  install -d "$pkgdir/usr/share/koboldcpp"
  cp -a . "$pkgdir/usr/share/koboldcpp"

  echo '#!/bin/sh' > koboldcpp
  echo 'cd /usr/share/koboldcpp && python koboldcpp.py "$@"' >> koboldcpp

  install -Dm755 koboldcpp "$pkgdir/usr/bin/koboldcpp"
}

ilikenwf commented on 2024-02-23 04:02 (UTC)

Customkinter and libtk need to be required. Also, version 1.58 is current.

Lucki commented on 2024-02-22 17:04 (UTC) (edited on 2024-02-22 17:11 (UTC) by Lucki)

I get the following error when loading a model:

ggml-cuda.cu:3019: ERROR: CUDA kernel vec_dot_q5_K_q8_1_impl_vmmq has no device code compatible with CUDA arch 520. ggml-cuda.cu was compiled for: 520
CUDA error: the function failed to launch on the GPU
  current device: 0, in function ggml_cuda_op_mul_mat_cublas at ggml-cuda.cu:8822
  cublasSgemm_v2(g_cublas_handles[id], CUBLAS_OP_T, CUBLAS_OP_N, row_diff, src1_ncols, ne10, &alpha, src0_ddf_i, ne00, src1_ddf1_i, ne10, &beta, dst_dd_i, ldc)
GGML_ASSERT: ggml-cuda.cu:241: !"CUDA error"

With the following patch I was able to successfully load a model:

diff --git a/Makefile b/Makefile
index 892f17e7..37d78ee4 100644
--- a/Makefile
+++ b/Makefile
@@ -160,7 +160,7 @@ else
    NVCCFLAGS += -Wno-deprecated-gpu-targets -arch=all
 endif #LLAMA_COLAB
 else
-   NVCCFLAGS += -arch=native
+   NVCCFLAGS += -arch=all
 endif #LLAMA_PORTABLE
 endif # CUDA_DOCKER_ARCH