Package Details: python-transformers 4.51.2-1

Git Clone URL: https://aur.archlinux.org/python-transformers.git (read-only, click to copy)
Package Base: python-transformers
Description: State-of-the-art Natural Language Processing for Jax, PyTorch and TensorFlow
Upstream URL: https://github.com/huggingface/transformers
Keywords: huggingface transformers
Licenses: Apache-2.0
Submitter: filipg
Maintainer: daskol
Last Packager: daskol
Votes: 15
Popularity: 0.51
First Submitted: 2021-10-23 09:30 (UTC)
Last Updated: 2025-04-10 22:06 (UTC)

Dependencies (23)

Sources (1)

Pinned Comments

daskol commented on 2025-04-07 08:09 (UTC) (edited on 2025-04-07 08:09 (UTC) by daskol)

NOTE The next v4.51.0 requires xet-core (transitive through python-huggingface-hub>=1:0.30.0). It is now under testing in extra-testing.

Latest Comments

« First ‹ Previous 1 2 3 4 Next › Last »

daskol commented on 2023-07-08 12:11 (UTC) (edited on 2023-07-08 12:15 (UTC) by daskol)

@ttc0419 Exactly. Most of dependencies of transformers are optional (see repo). By design HuggingFace is aimed at major deep learning frameworks TF, PT, JAX but the issue is that a user usually adhere to a greater extent to a single framework. So, enumeration of only required dependencies gives a user a freedom to manage installed packages on user's system in more fine-grained way.

The errors I listed are NOT optional.

They are actually optional. Just check transformers repo. It is a nightmare actually. The list of actual dependencies depends on a framework and model. In general it is impossible to manage this kind of projects properly.

ttc0419 commented on 2023-07-08 12:04 (UTC) (edited on 2023-07-08 12:07 (UTC) by ttc0419)

@xiota Why avoiding "listing hundreds of packages"? One purpose of the package is dependency management. It's not something nice to let users to track down decencies themselves. If it's optional, list it as optional. The errors I listed are NOT optional. It occurs immediately after an import, which is required packages missing. BTW, you can refer to pip for the dependencies of a package, the list of transformers is actually not very long:

transformers==4.31.0.dev0
├── filelock [required: Any, installed: 3.12.2]
├── huggingface-hub [required: >=0.14.1,<1.0, installed: 0.16.3]
│   ├── filelock [required: Any, installed: 3.12.2]
│   ├── fsspec [required: Any, installed: 2023.6.0]
│   ├── packaging [required: >=20.9, installed: 23.1]
│   ├── PyYAML [required: >=5.1, installed: 6.0]
│   ├── requests [required: Any, installed: 2.31.0]
│   │   ├── certifi [required: >=2017.4.17, installed: 2023.5.7]
│   │   ├── charset-normalizer [required: >=2,<4, installed: 3.1.0]
│   │   ├── idna [required: >=2.5,<4, installed: 3.4]
│   │   └── urllib3 [required: >=1.21.1,<3, installed: 2.0.3]
│   ├── tqdm [required: >=4.42.1, installed: 4.65.0]
│   │   └── colorama [required: Any, installed: 0.4.6]
│   └── typing-extensions [required: >=3.7.4.3, installed: 4.7.1]
├── numpy [required: >=1.17, installed: 1.25.0]
├── packaging [required: >=20.0, installed: 23.1]
├── PyYAML [required: >=5.1, installed: 6.0]
├── regex [required: !=2019.12.17, installed: 2023.6.3]
├── requests [required: Any, installed: 2.31.0]
│   ├── certifi [required: >=2017.4.17, installed: 2023.5.7]
│   ├── charset-normalizer [required: >=2,<4, installed: 3.1.0]
│   ├── idna [required: >=2.5,<4, installed: 3.4]
│   └── urllib3 [required: >=1.21.1,<3, installed: 2.0.3]
├── safetensors [required: >=0.3.1, installed: 0.3.1]
├── tokenizers [required: >=0.11.1,<0.14,!=0.11.3, installed: 0.13.3]
└── tqdm [required: >=4.27, installed: 4.65.0]
    └── colorama [required: Any, installed: 0.4.6]

ttc0419 commented on 2023-07-08 06:13 (UTC)

Multiple missing dependencies:

ImportError: 
LlamaTokenizer requires the SentencePiece library but it was not found in your environment. Checkout the instructions on the
installation page of its repo: https://github.com/google/sentencepiece#installation and follow the ones
that match your environment. Please note that you may need to restart your runtime after installation.

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1086, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 31, in <module>
    from ...modeling_utils import PreTrainedModel
  File "/usr/lib/python3.11/site-packages/transformers/modeling_utils.py", line 40, in <module>
    from .pytorch_utils import (  # noqa: F401
  File "/usr/lib/python3.11/site-packages/transformers/pytorch_utils.py", line 19, in <module>
    from safetensors.torch import storage_ptr, storage_size
ModuleNotFoundError: No module named 'safetensors'

daskol commented on 2023-03-24 11:50 (UTC) (edited on 2023-03-24 11:50 (UTC) by daskol)

@AllesMeins @DeathByDenim It was a bug in setuptools (see issue)

daskol commented on 2023-03-24 11:47 (UTC)

@DeathByDenim Did you try to update to update to the latest python-setuptools (1:67.5.1-1 by the momemnt)? Arch is rolling release distro still.

DeathByDenim commented on 2023-03-24 11:44 (UTC)

@daskol I have the same issue as AllesMeins. My version of setuptools is 67.3.0.

daskol commented on 2023-03-22 22:04 (UTC)

@AllesMeins I've checked package building in isolated environment in docker container. Everything is fine.

What version of setuptools do you use? It seems that it is obsolete.

AllesMeins commented on 2023-03-22 21:43 (UTC) (edited on 2023-03-22 21:44 (UTC) by AllesMeins)

Building of the latest version fails for me. Previous version was building fine:

==> Starting build()...
* Getting build dependencies for wheel...
running egg_info
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
  File "/usr/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
  File "/usr/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
    self.run_setup()
  File "/usr/lib/python3.10/site-packages/setuptools/build_meta.py", line 484, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/usr/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
    exec(code, locals())
  File "<string>", line 419, in <module>
  File "/usr/lib/python3.10/site-packages/setuptools/__init__.py", line 108, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/usr/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
    super().run_command(command)
  File "/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
    self.finalize_options()
  File "/usr/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 253, in finalize_options
    if pd is not None and pd.key == self.egg_name.lower():
AttributeError: 'PathDistribution' object has no attribute 'key'

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel
==> ERROR: A failure occurred in build().

« First ‹ Previous 1 2 3 4 Next › Last »