Package Details: llama.cpp-cuda b10092-1

Git Clone URL: https://aur.archlinux.org/llama.cpp-cuda.git (read-only, click to copy)
Package Base: llama.cpp-cuda
Description: Port of Facebook's LLaMA model in C/C++ (with NVIDIA CUDA optimizations)
Upstream URL: https://github.com/ggml-org/llama.cpp
Licenses: MIT
Conflicts: ggml, libggml, llama.cpp
Provides: llama.cpp
Submitter: txtsd
Maintainer: fabse
Last Packager: fabse
Votes: 22
Popularity: 2.97
First Submitted: 2024-10-26 20:17 (UTC)
Last Updated: 2026-07-23 06:32 (UTC)

Dependencies (19)

Required by (13)

Sources (3)

Pinned Comments

txtsd commented on 2024-10-26 20:17 (UTC) (edited on 2024-12-06 14:15 (UTC) by txtsd)

Alternate versions

llama.cpp
llama.cpp-vulkan
llama.cpp-sycl-fp16
llama.cpp-sycl-fp32
llama.cpp-cuda
llama.cpp-cuda-f16
llama.cpp-hip

Latest Comments

« First ‹ Previous 1 .. 5 6 7 8 9 10 11 Next › Last »

v1993 commented on 2024-12-01 14:53 (UTC)

To be honest, I'm not 100% sure (it's a pretty old option and tacking down its origins is kinda tricky), but I'd expect at least a performance degradation on older GPUs (Nvidia used to be really bad at fp16 on older architectures).

txtsd commented on 2024-12-01 14:38 (UTC)

@v1993 Does that have to be a separate package, or will making the change in this package suffice without breaking things for users of older GPUs?

v1993 commented on 2024-12-01 14:29 (UTC)

Would it be possible to have a package version with GGML_CUDA_F16 enabled? It's a nice performance boost on newer GPUs. Thank you for your work on this package!

Poscat commented on 2024-11-28 09:46 (UTC)

@txtsd thank you

txtsd commented on 2024-11-25 07:05 (UTC)

Builds are not static anymore, and the service file has been fixed.

txtsd commented on 2024-11-24 03:16 (UTC)

@Poscat Thank you for your input! The service file was inherited from a previous version and maintainer of the package. I admit that the service was not tested.

The static builds were created to allow for side-by-side installation with whisper.cpp, since they both install libggml files.

Poscat commented on 2024-11-24 03:12 (UTC) 

diff --git a/llama.cpp.service b/llama.cpp.service
index 4678d85..be89f9b 100644
--- a/llama.cpp.service
+++ b/llama.cpp.service
@@ -7,7 +7,7 @@ Type=simple
 EnvironmentFile=/etc/conf.d/llama.cpp
 ExecStart=/usr/bin/llama-server $LLAMA_ARGS
 ExecReload=/bin/kill -s HUP $MAINPID
-Restart=never
+Restart=no

 [Install]
 WantedBy=multi-user.target

Also your sysetmd service file is wrong. Did you even test your package?

Poscat commented on 2024-11-24 03:10 (UTC) (edited on 2024-11-24 03:10 (UTC) by Poscat) 

diff --git a/PKGBUILD b/PKGBUILD
index ad448a7..3fdc20f 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -50,7 +50,8 @@ build() {
   local _cmake_options=(
     -B build
     -S "${_pkgname}"
-    -DCMAKE_BUILD_TYPE=None
+    -DCMAKE_BUILD_TYPE=MinSizeRel
+    -DCMAKE_CUDA_ARCHITECTURES=native
     -DCMAKE_INSTALL_PREFIX='/usr'
     -DGGML_NATIVE=OFF
     -DGGML_AVX2=OFF
@@ -59,8 +60,8 @@ build() {
     -DGGML_FMA=OFF
     -DGGML_ALL_WARNINGS=OFF
     -DGGML_ALL_WARNINGS_3RD_PARTY=OFF
-    -DBUILD_SHARED_LIBS=OFF
-    -DGGML_STATIC=ON
+    -DBUILD_SHARED_LIBS=ON
+    -DGGML_STATIC=OFF
     -DGGML_LTO=ON
     -DGGML_RPC=ON
     -DLLAMA_CURL=ON
@@ -75,7 +76,6 @@ build() {
 package() {
   DESTDIR="${pkgdir}" cmake --install build
   rm "${pkgdir}/usr/include/"ggml*
-  rm "${pkgdir}/usr/lib/"lib*.a

   install -Dm644 "${_pkgname}/LICENSE" "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"

this patch reduces the package size from 37G to 82M

Poscat commented on 2024-11-24 01:55 (UTC)

Maybe don't enable static linking? IDK

Poscat commented on 2024-11-24 01:54 (UTC) (edited on 2024-11-24 01:54 (UTC) by Poscat) 

❯ ls -lh /usr/bin/llama*
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-batched
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-batched-bench
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-bench
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-cli
-rwxr-xr-x 1 root root 814M Nov 23 00:02 /usr/bin/llama-convert-llama2c-to-ggml
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-cvector-generator
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-embedding
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-eval-callback
-rwxr-xr-x 1 root root 814M Nov 23 00:02 /usr/bin/llama-export-lora
-rwxr-xr-x 1 root root 814M Nov 23 00:02 /usr/bin/llama-gbnf-validator
-rwxr-xr-x 1 root root 111K Nov 23 00:02 /usr/bin/llama-gguf
-rwxr-xr-x 1 root root 131K Nov 23 00:02 /usr/bin/llama-gguf-hash
-rwxr-xr-x 1 root root 814M Nov 23 00:02 /usr/bin/llama-gguf-split
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-gritlm
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-imatrix
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-infill
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-llava-cli
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-lookahead
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-lookup
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-lookup-create
-rwxr-xr-x 1 root root  31K Nov 23 00:02 /usr/bin/llama-lookup-merge
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-lookup-stats
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-minicpmv-cli
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-parallel
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-passkey
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-perplexity
-rwxr-xr-x 1 root root 814M Nov 23 00:02 /usr/bin/llama-quantize
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-quantize-stats
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-retrieval
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-save-load-state
-rwxr-xr-x 1 root root 817M Nov 23 00:02 /usr/bin/llama-server
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-simple
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-simple-chat
-rwxr-xr-x 1 root root 816M Nov 23 00:02 /usr/bin/llama-speculative
-rwxr-xr-x 1 root root 815M Nov 23 00:02 /usr/bin/llama-tokenize

« First ‹ Previous 1 .. 5 6 7 8 9 10 11 Next › Last »