Package Details: python-tokenizers 0.19.1-2

Git Clone URL: (read-only, click to copy)
Package Base: python-tokenizers
Description: Fast State-of-the-Art Tokenizers optimized for Research and Production
Upstream URL:
Keywords: huggingface
Licenses: Apache-2.0
Submitter: filipg
Maintainer: xiota (daskol)
Last Packager: daskol
Votes: 6
Popularity: 1.09
First Submitted: 2021-10-23 11:17 (UTC)
Last Updated: 2024-05-07 14:02 (UTC)

Pinned Comments

xiota commented on 2024-02-10 21:08 (UTC)

This package has been successfully built in a clean chroot.

Latest Comments

1 2 3 4 Next › Last »

xiota commented on 2024-02-19 00:26 (UTC) (edited on 2024-02-19 00:27 (UTC) by xiota)

@rekman There is nothing to fix. You have not claimed or provided any logs to demonstrate any problem that cannot be resolved by building in a clean chroot or makepkg -C. There is no helping users who are too "lazy" to type two extra characters.

If you do not provide evidence of an issue that cannot be solved by building in a clean chroot, this is the end of discussion on this topic.

rekman commented on 2024-02-19 00:08 (UTC)

@xiota it is in the packaging guidelines but, sure, make things more difficult for users for no reason at all except being lazy or an asshole.

xiota commented on 2024-02-18 23:15 (UTC)

@rekman I am not adding cleanup commands to the PKGBUILD. That is user responsibility. Use makepkg -C.

rekman commented on 2024-02-18 23:05 (UTC)

@xioto it solves the problem of always having to manually build in a clean chroot......?

xiota commented on 2024-02-10 21:08 (UTC)

This package has been successfully built in a clean chroot.

edtoml commented on 2024-02-10 20:01 (UTC)

This is the second package in as many days that is broken because it requires the nightly version of rust.

xiota commented on 2024-02-10 05:23 (UTC)

@rekman Is there a real problem that is intended to solve? Why aren't the following sufficient?

rekman commented on 2024-02-10 00:45 (UTC)

Recommend adding git -C "${srcdir}/${pkgname}" clean -dfx to prepare() to clean out stale wheels.

xiota commented on 2024-02-09 02:14 (UTC)

@dreieck Thank you for finding a solution to the oniguruma issue.

I am not removing the --locked option because this package builds fine in a clean chroot. It is also recommended by rust package guidelines.

I am not adding the --offline option because there is no benefit. It would only break the package if cargo needs to download something.

dreieck commented on 2024-02-08 15:58 (UTC) (edited on 2024-02-08 16:20 (UTC) by dreieck)

Adding options+=('!lto') fixes the

ImportError: /usr/lib/python3.11/site-packages/tokenizers/ undefined symbol: OnigDefaultSyntax

issue for me.