Package Details: python-safetensors 0.4.3-2

Git Clone URL: https://aur.archlinux.org/python-safetensors.git (read-only, click to copy)
Package Base: python-safetensors
Description: Simple, safe way to store and distribute tensors
Upstream URL: https://github.com/huggingface/safetensors
Licenses: Apache
Submitter: Henry-ZHR
Maintainer: mane.andrea
Last Packager: mane.andrea
Votes: 2
Popularity: 0.92
First Submitted: 2023-01-08 09:54 (UTC)
Last Updated: 2024-04-17 17:56 (UTC)

Sources (1)

Latest Comments

1 2 3 Next › Last »

mane.andrea commented on 2024-04-17 17:57 (UTC)

@carsme Many thanks, I have patched it as you say

carsme commented on 2024-04-17 17:51 (UTC)

Two issues:

  • Re-building without cleaning the build directory fails due to the mkdir invocation.
  • Building with rustup instead of rust fails, due to that no toolchain version is specified.

These issues can be fixed by the following patch:

diff --git a/PKGBUILD b/PKGBUILD
index 03774ec..e069144 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -39,11 +39,13 @@ source=("${pkgname}-${pkgver}.tar.gz::${url}/archive/refs/tags/v${pkgver}.tar.gz
 sha512sums=('278469d7bc8a4285519cf8020891d0f0e107d8124ad2c24686ca8a19ef2f210c49d9483c502bd5aa5d73940a775ac3eb5e3b90f62825dd7845c351bbdc8c82c3')

 prepare() {
-  mkdir "safetensors-${pkgver}/.git"
+  mkdir -p "safetensors-${pkgver}/.git"
 }

 build() {
   cd "safetensors-${pkgver}/bindings/python"
+
+  export RUSTUP_TOOLCHAIN=stable
   python -m build --wheel --no-isolation
 }

mane.andrea commented on 2023-12-12 18:29 (UTC)

With @daskol patch now builds fine. I tried a chroot build in a machine with cuda and in a machine without cuda

mane.andrea commented on 2023-12-09 03:42 (UTC)

Sorry for the delay. I overlooked the notification. Will work on an updated PKGBUILD asap

daskol commented on 2023-12-09 02:41 (UTC)

In my optinion, the best solution is not put any .gitignore with glob pattern in AUR repo since there is no actual need in blacklisting files at all.

@h3ss I believe that it'd be better to create an empty directory .git at $srcdir/safetensors-$pkgver rather than patching or sed'ing (see ignore::WalkBuilder). Anyway, we should create an issue in PyO3/maturin. It is stupid that it traverses out of repo root. What about submodules? Why I should tweak include/ignore filters in pyproject.toml? Weird.

diff --git a/PKGBUILD b/PKGBUILD
index cb94dc8..4d04fcf 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -38,6 +38,10 @@ checkdepends=('python-pytorch'
 source=("${pkgname}-${pkgver}.tar.gz::${url}/archive/refs/tags/v${pkgver}.tar.gz")
 sha512sums=('da38f184f2c0730a59b1d4ca490758b71cb5ed3e1e16741bbf4b7bfac35fa1a4f24e5acbc35f3b38f10e4c46d81554896fbd95accfd671ee066c9fcbd50551ab')

+prepare() {
+  mkdir "safetensors-${pkgver}/.git"
+}
+
 build() {
   cd "safetensors-${pkgver}/bindings/python"
   python -m build --wheel --no-isolation

@xiota There is the same issue with python-tokenziers.

h3ss commented on 2023-12-05 03:34 (UTC)

The problem is that the AUR package includes a .gitignore file, and the maturin build backend will ignore anything specified in any .gitignore file in the path.

See: - Wheels for mixed Python/Rust project does not include Python source code #885

The contents of the AUR .gitignore are:

**/*
!.SRCINFO
!.gitignore
!PKGBUILD

That first line is the problem, it basically excludes everything from inclusion (except what the Rust part of the build creates apparently).

I have been able to get around this by editing the src/safetensors-0.4.0/bindings/python/pyproject.toml file. Specifically, by adding this under the [tool.maturin] section:

include = ["**/*"]

See: - Maturin User Guide - Configuration - Specifically the include part

When this is done, the resulting .whl file reliably contains the needed Python source files.

So, how can we get around this in the PKGBUILD? By using sed to add the include line to the pyproject.toml in the build() function:

build() {
  cd "safetensors-${pkgver}/bindings/python"
  sed -i 's/\[tool.maturin\]/[tool.maturin]\ninclude = \["**\/*"\]/g' pyproject.toml
  python -m build --wheel --no-isolation
}

After adding that sed line, the package reliably passes all checks and builds.

@mane.andrea Can we get this or something equivalent incorporated into the PKGBUILD? I'm confident this should fix this longstanding issue.

PhotonX commented on 2023-11-26 18:14 (UTC) (edited on 2023-11-26 18:15 (UTC) by PhotonX)

I'm getting the check() errors as well. I don't understand what the underlying problem is, though. One of the errors is, exemlarily:

________________ ERROR collecting tests/test_flax_comparison.py ________________
ImportError while importing test module '/home/photon/.cache/yay/python-safetensors/src/safetensors-0.4.0/bindings/python/tests/test_flax_comparison.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_flax_comparison.py:12: in <module>
    from safetensors import safe_open
E   ImportError: cannot import name 'safe_open' from 'safetensors' (unknown location)

So it searches for safe_open from... itself? I think, I have all the necessary dependencies installed, but none of them provides this...

t3nzor commented on 2023-11-20 02:22 (UTC) (edited on 2023-11-20 02:22 (UTC) by t3nzor)

The built package doesn't seem to install the Python bindings, only the shared library. I was able to work around this as follows:

makepkg -si --nocheck

sudo cp src/safetensors-0.4.0/bindings/python/py_src/safetensors/* /usr/lib/python3.11/site-packages/safetensors

65a commented on 2023-11-18 02:52 (UTC) (edited on 2023-11-18 02:59 (UTC) by 65a)

@phoenix17, I'm glad it works for you, but it doesn't work for me. The result is still failing to import safetensors.torch or crashing in the check() (this depends on which machine I use?!), using a clean git clone of the AUR source then makepkg -si on multiple machines. Here is one failure log: https://pastebin.com/mppuQw0H

phoenix17 commented on 2023-11-16 20:07 (UTC)

In case this helps someone, I was able to build/install this manually via makepkg -si without making any changes to the build script here: https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=python-safetensors

I can confirm it also worked with python-transformers package on the aur and pytorch (https://archlinux.org/packages/extra/x86_64/python-pytorch/)

1 2 3 Next › Last »