Arch Linux User Repository

You should determine the sitelib path dynamically:

https://wiki.archlinux.org/title/Python_package_guidelines

@gnaggnoyil, fixed. Thanks for pointing this out.

Current version of core/python is 3.10 while PYTHONPATH is set to /usr/lib/python3.9 in check of PKGBUILD. I think either PYTHONPATH needs to be updated to match current core/python version or a python=3.9 dependency needs to be explicitly specified in the dependencies section.

Can you please add python-hypothesis to the makedepends array. It's required for the check function. Thanks!

Yes a distribution-wide switch would be ideal (Arch does it for Java, and others like Debian have a generic method that they use for BLAS too).

Hi,

the mentioned ArchWiki page and an Archlinux wide switching approach would be greatly appreciated.

See also https://github.com/python-control/Slycot/issues/138#issuecomment-703573577

I like this package because it just works. It removes all the conflicting packages, so please keep this package

I was thinking about creating an ArchWiki page that explains what is BLAS & LAPACK, which packages use them, what are the available providers, and how to switch. This could be linked from the user package wikis (numpy, R, julia, etc.). Would that solve the issue in your opinion?

@mdeff That sounds great.

@xia0er, thanks for your consideration. Isn't it overkill to have to rebuild the package for an additional dependency? Also, such a policy quickly explodes, as you'd need every <package>-<blas> combinations with {numpy, scipy, julia, r, pytorch, tensorflow, opencv, arpack, etc.} as packages and {netlib, openblas, mkl, blis, atlas} as blas. I think it's cleaner for user packages to depend on {blas, cblas, lapack, lapacke} and for provider packages to provide those. Users then choose which provider to install.

I was thinking about creating an ArchWiki page that explains what is BLAS & LAPACK, which packages use them, what are the available providers, and how to switch. This could be linked from the user package wikis (numpy, R, julia, etc.). Would that solve the issue in your opinion?

In the longer term, Arch should ideally allow the installation of multiple providers and have a mechanism to easily switch the default, like Debian, Gentoo, conda-forge (Arch does it for java).

Do you remember where the discussion about not including LAPACK in community/openblas happened? I searched as much as I could and didn't find any. In my opinion it's a serious issue (most serious being the omission of the C bindings), and the maintainer doesn't seem responsive. :/

@mdeff Just checked out the FS#s you linked to. If the community/openblas eventually includes lapack, I agree this package can be sunset/removed, but I'd like to keep it active before that.

@mdeff, the whole purpose of this package is to specify the dependency on aur/openblas-lapack. The community/openblas doesn't provide lapack (and will not, as I recall from an early discussion in the openblas package and why the aur/openblas-lapack package was created). As you pointed out, one can install aur/openblas-lapack and community/python-numpy, and have an optimized bumpy installation, but there is no clear way to specify the dependency than this package.

You can check what numpy is actually dynamically linked to with ldd /usr/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so and ldd /usr/lib/python3.8/site-packages/numpy/linalg/_umath_linalg.cpython-38-x86_64-linux-gnu.so. There will be a line about BLAS, like libblas.so.3 => /usr/lib/libblas.so.3 and a line about LAPACK, like liblapack.so.3 => /usr/lib/liblapack.so.3. Numpy links to the system-provided BLAS and LAPACK. We can check where they come from.

1) If you have extra/blas, extra/cblas, and extra/lapack installed (i.e., BLAS and LAPACK from netlib), /usr/lib/libblas.so.3 will be a symlink to /usr/lib/libblas.so.3.9.0 (owned by extra/blas), and /usr/lib/liblapack.so.3 will be a symlink to /usr/lib/liblapack.so.3.9.0 (owned by extra/lapack).

2) If you have community/openblas, extra/cblas, and extra/lapack installed (i.e., BLAS from OpenBLAS and LAPACK from netlib), /usr/lib/libblas.so.3 will be a symlink to /usr/lib/libopenblasp-r0.3.9.so (owned by community/openblas) and /usr/lib/liblapack.so.3 will be a symlink to /usr/lib/liblapack.so.3.9.0 (owned by extra/lapack). (FYI, I don't recommended this configuration. See FS#66092, FS#59046, FS#63054, FS#62558. The community/openblas package should be fixed and could replace aur/openblas-lapack.)

3) If you have aur/openblas-lapack installed (i.e., BLAS and LAPACK from OpenBLAS), /usr/lib/libblas.so.3 and /usr/lib/liblapack.so.3 will be symlinks to /usr/lib/libopenblas.so (owned by aur/openblas-lapack).

4) You can actually symlink any BLAS and LAPACK provider (e.g., MKL, ATLAS, BLIS) to those locations, and numpy and scipy, (but also R, Julia, Octave, etc.) will use them. That's because BLAS and LAPACK are defined as APIs. Those are implemented by multiple providers which can be dynamically loaded at runtime.

@mdeff Is there a better way to verify it instead of timing it?

As @adfjjv pointed out, the stock extra/python-numpy will use OpenBLAS if installed (by community/openblas or aur/openblas-lapack).

You can verify it empirically:

sudo pacman -S blas lapack cblas python-numpy  # BLAS and LAPACK from netlib
time python -c "import numpy as np; x=np.random.random((2000, 2000)); np.dot(x, x.T)"
# about 4 seconds on my system, single thread
sudo pacman -S openblas lapack cblas python-numpy  # BLAS from OpenBLAS, LAPACK from netlib
time python -c "import numpy as np; x=np.random.random((2000, 2000)); np.dot(x, x.T)"
# about 0.3 second on my system, multiple threads
sudo pacman -S openblas-lapack python-numpy  # BLAS and LAPACK from OpenBLAS
time python -c "import numpy as np; x=np.random.random((2000, 2000)); np.dot(x, x.T)"
# about 0.3 second on my system, multiple threads

I suggest to remove this package to avoid further confusion (and aur/python-scipy-openblas).

@xia0er. I get this issue while I try to re-install (doing a clean build).

Edit: Solved after re-updating openblas-lapack!

@masca, re-installing python-numpy-openblas should resolve your issue.

After the update of the openblas-lapack packages, NumPy does not work:

Original error was: libopenblas.so.3: cannot open shared object file: No such file or directory

numpy.test() seems to get stuck at 35%.

@adfjjv, the purpose of the package is exactly as you described - building and installing python-numpy after openblas-lapack. The stock python-numpy installed with pacman isn't linked to openblas-lapack.

What is the purpose of this package? I mean, doesn't installing regular python-numpy together with openblas-lapack work?

@aviallon, good to know. Thanks for reporting back and hope it'd be helpful to others.

@xia0er Hello, finally found my problem, after a VERY long time : it was caused by packages enum34 ! I just had to uninstall it and voilà ! See https://stackoverflow.com/a/45716067/6061951 for more info.

@xiaOer : well, it happens at the same moment, but I did not have anything weird in my PATH.

aviallon, I cannot replicate your problem. Is your problem related to what is described here: https://github.com/numpy/numpy/issues/6917

I am getting RuntimeError:("Running cythonyze failed!") now. On two different systems, both with last cython version installed (even tried the git version, with no success).

@thrasibule we now converted to use split package for both. Also, I follow the python-numpy and rename f2py installed by python2-numpy-openblas. They two can now be installed side-by-side.

Same issue with 1.16. python2-numpy-openblas conflicts with python-numpy-openblas as they both install f2py.

This package doesn't coinstall with python-numpy-openblas. Also any reason why not make python-numpy-openblas a split package which would install both?

python-numpy-openblas and python2-numpy-openblas can't be installed together anymore. /usr/bin/conv-template, /usr/bin/from-template and /usr/bin/f2py exist in both packages. Also any reason why not make it a split package and drop the python2-numpy-openblas package?

@Pastafarianist From the discussions on the linked sklearn issue page, it doesn't seem to be an issue with python-numpy-openblas, as it also occurs with numpy-mkl and pip installed numpy? Hope you'll be able to diagnose the cause and come up with a fix soon.

@solnce: You are correct. The mistake was on my part. After recompiling the OpenBLAS library 3 times from source, I realized that I was using the openblas-lapack package provided by AUR on this computer. Recompiling in the same order you did works totally fine. Apologies for any inconveniences my previous message may have caused.

@domochevski: I just recompiled it, works fine for me. The new packages from this morning might require a more thorough recompile. I did it in this order: (1) openblas-lapack, (2) python-numpy-openblas, (3) python-scipy-openblas.

It seems like the latest update of the gcc-libs package is causing the whole compile process to crash, making it impossible to build and install this package.

Importing sklearn with python-numpy-openblas (or python-numpy-mkl) installed causes CPU affinity to be reset to {0}. Please see https://github.com/scikit-learn/scikit-learn/issues/11071 for more info and a PoC. The issue does not manifest with python-numpy.

@sakel: You can downgrade to the penultimate version of python-matplotlib, which solved it for me.

I'm contacting xia0er by mail now. If there is no answer by 2018-02-14, I will file an orphan request.

and 1.13.1 causes python-matplotlib to crash :'(

1.14 is out :)