Package Details: python-tokenizers 0.14.0-1

Git Clone URL: https://aur.archlinux.org/python-tokenizers.git (read-only, click to copy)
Package Base: python-tokenizers
Description: Fast State-of-the-Art Tokenizers optimized for Research and Production
Upstream URL: https://github.com/huggingface/tokenizers
Licenses: Apache
Conflicts: python-tokenizers
Provides: python-tokenizers
Submitter: filipg
Maintainer: xiota
Last Packager: xiota
Votes: 3
Popularity: 0.165858
First Submitted: 2021-10-23 11:17 (UTC)
Last Updated: 2023-09-08 07:53 (UTC)

Latest Comments

1 2 Next › Last »

xiota commented on 2023-05-30 05:02 (UTC) (edited on 2023-09-08 07:28 (UTC) by xiota)

@dreieck Updated so that some crates are downloaded in prepare(). More crates are downloaded midway through the build process. This is out of my control because the build process is controlled by python scripts. You'll have to work with upstream if you want this changed.

dreieck commented on 2023-05-04 09:54 (UTC)

This PKGBUILD downloads stuff during build() and stores that in the user's $HOME directory ($HOME/.cargo/).

Can you

  1. make sure that all the rust dependency downloads take place in prepare(), so that build() and package() do not need internet connection, and
  2. make sure that the download goes into some subdirectory of $srcdir, to not clutter the user's home directory (I think $CARGO_HOME is the environment variable that controls this, but please cross-check for yourself, also with python-setuptools-rust specifities)?
cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib
    Updating crates.io index
       Fetch [                         ]   1.81%, 295.81KiB/s

Regards and
thanks for maintaining!

taba commented on 2023-02-27 20:07 (UTC)

Ignore what I said. I think I was being pedantic. Sorry for the notification.

xiota commented on 2023-02-27 18:45 (UTC) (edited on 2023-02-27 18:46 (UTC) by xiota)

Why? The release is tagged. This way is easier to update versions or switch to a git build.

taba commented on 2023-02-27 18:08 (UTC)

Pin the source release hash in PKGBUILD. Use https://github.com/huggingface/tokenizers/archive/refs/tags/v0.13.2.tar.gz.

xiota commented on 2023-02-14 03:02 (UTC)

@groo This package should build in a clean chroot (extra-x86_64-build). If you're having problems, try running makepkg directly instead of using AUR helpers.

groo commented on 2022-12-16 18:09 (UTC)

The package does not compile for me. It fails with following message:

[...output ommitted...] running build_rust cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib error: 'json-render-diagnostics' isn't a valid value for '--message-format <FMT>' [possible values: human, json, short]

    Did you mean 'json'?

USAGE: cargo rustc --features <FEATURES> --lib --manifest-path <PATH> --message-format <FMT> --release --verbose

For more information try --help error: cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib failed with code 1 ==> FEHLER: Ein Fehler geschah in build(). Breche ab... -> Fehler beim Erstellen: python-tokenizers

filipg commented on 2022-12-10 08:07 (UTC)

It depends on python-setuptools-rust which, in turn, depends on python-setuptools, so I assume it is not needed to add it explicitly.

1 2 Next › Last »