Package Details: python-tokenizers 0.21.1-1

Git Clone URL: https://aur.archlinux.org/python-tokenizers.git (read-only, click to copy)
Package Base: python-tokenizers
Description: Fast State-of-the-Art Tokenizers optimized for Research and Production
Upstream URL: https://github.com/huggingface/tokenizers
Keywords: huggingface
Licenses: Apache-2.0
Submitter: filipg
Maintainer: xiota (daskol)
Last Packager: daskol
Votes: 9
Popularity: 0.45
First Submitted: 2021-10-23 11:17 (UTC)
Last Updated: 2025-03-17 18:15 (UTC)

Pinned Comments

xiota commented on 2024-08-30 16:15 (UTC) (edited on 2024-08-30 16:59 (UTC) by xiota)

Problems:

Latest Comments

1 2 3 4 Next › Last »

envolution commented on 2025-05-03 04:54 (UTC) (edited on 2025-05-03 04:55 (UTC) by envolution)

I was getting an error from the build process of cargo package oniguruma/onig_sys v69.8.1 related to incompatible void pointers. I was able to work around it by adding export CFLAGS="$CFLAGS -std=gnu89" just before cargo build and cargo test. There's probably a better solution but this was fine for my purposes.

carlosal1015 commented on 2025-01-21 21:19 (UTC)

https://github.com/huggingface/tokenizers/blob/v0.21.0/bindings/python/benches/test_tiktoken.py#L10

python-multiprocess is now in [extra].

envolution commented on 2024-12-13 05:48 (UTC)

@xiota can you please bump the version?

[I 12-13 00:46:46.750 core:416] python-tokenizers: updated to 0.21.0

rekman commented on 2024-08-30 22:04 (UTC)

Trying and struggling to understand the mindset of the person who would rather keep typing out justifications for not improving the experience of downstream users rather than pasting one goddamn line into a file and pressing push. Like you can either keep telling every user who gets an extremely confusing error message to "try building in a clean chroot" forever, or you can paste one fucking line and hit push. No, AUR helpers should not need to assume that every package needs to be built in a clean chroot, that's a huge waste of time for any upgrade when in fact most every other package on the AUR manages to build fine incrementally, and the solution here is trivial. Furthermore, while this is not a VCS package the tip applies equally because it is still built locally rather than just having a wheel downloaded. Leads to the exact same situation What's the problem anyway? Some purity bullshit or what? What the hell do you think the prepare() function is for to begin with? You are the kind of person that turns people away from FOSS.

xiota commented on 2024-08-30 16:15 (UTC) (edited on 2024-08-30 16:59 (UTC) by xiota)

Problems:

rekman commented on 2024-08-30 15:22 (UTC) (edited on 2024-08-30 15:23 (UTC) by rekman)

Yes, I am lazy, as ar many other users, but not as lazy as the person who won't add a line that doesn't even have to be typed, just PASTED. You can either add one fucking line to the PKGBUILD or you can continue to bloviate about how it's the user's fault that your PKGBUILD does not conform to packaging guidelines and doesn't play well with automated AUR helpers. One seems like more effort than the other to me, but what do I know, I don't have a gigantic self-righteous prideful stick up my ass.

xiota commented on 2024-02-19 00:26 (UTC) (edited on 2024-02-19 00:27 (UTC) by xiota)

@rekman There is nothing to fix. You have not claimed or provided any logs to demonstrate any problem that cannot be resolved by building in a clean chroot or makepkg -C. There is no helping users who are too "lazy" to type two extra characters.

If you do not provide evidence of an issue that cannot be solved by building in a clean chroot, this is the end of discussion on this topic.

rekman commented on 2024-02-19 00:08 (UTC)

@xiota it is in the packaging guidelines but, sure, make things more difficult for users for no reason at all except being lazy or an asshole.

xiota commented on 2024-02-18 23:15 (UTC)

@rekman I am not adding cleanup commands to the PKGBUILD. That is user responsibility. Use makepkg -C.

rekman commented on 2024-02-18 23:05 (UTC)

@xioto it solves the problem of always having to manually build in a clean chroot......?

1 2 3 4 Next › Last »