Package Details: cado-nfs-git 20210806.c5b20eac1-2

Git Clone URL: https://aur.archlinux.org/cado-nfs-git.git (read-only, click to copy)
Package Base: cado-nfs-git
Description: Implementation of the Number Field Sieve (NFS) algorithm for factoring integers
Upstream URL: http://cado-nfs.gforge.inria.fr/
Licenses: LGPL2
Conflicts: cado-nfs
Provides: cado-nfs
Submitter: jdetrey
Maintainer: AquilaIrreale
Last Packager: AquilaIrreale
Votes: 2
Popularity: 0.047031
First Submitted: 2015-12-07 12:57
Last Updated: 2021-08-15 13:53

Dependencies (12)

Required by (0)

Sources (1)

Latest Comments

1 2 Next › Last »

ccorn commented on 2021-08-15 23:16

From what I get to see of CMake files like avx2.cmake, the configuration goes as follows:

  1. Try to compile code for specific SIMD datatypes like avx2.c without adding own compiler flags.
  2. If compilation fails, check for -march.
  3. If no -march is specified, try adding flags like -mavx2.
  4. If compilation succeeds, try running the program on the build machine.
  5. If compilation or running fails, clear/unset the associated configuration variable like HAVE_AVX2.
  6. If running succeeds, set the associated configuration variable.

Thus, Haswell users should find HAVE_AVX2 defined in their cado_config.h, but Sandybridge users not.

To conclude, both the specified -march and the capabilities of the build system limit the available SIMD features: You cannot activate code paths meant for a target architecture more recent than that of your build system.

Worse, if you specify an architecture that is incompatible with your build system, then those incompatibilities may lead to failure of the autodetection of even those SIMD features that your build machine actually supports.

Thus, the only safe choices are

  • A minimum architecture that both build and target system can be expected to support, e.g. -march=sandybridge,
  • -march=native for the feature set of the build machine,
  • equivalently, no -march at all. Compared to -march=native, this adds more detailed compiler flags, but the HAVE_* definitions are identical.

ccorn commented on 2021-08-15 19:07

I wonder why _march=haswell does not result in the definition of HAVE_AVX2 in (my) cado_config.h. Does anyone achieve HAVE_AVX2 with _march=native?

ccorn commented on 2021-08-15 18:48

Anyhow, this is the resulting cado_config.h, looking about right I think

Yes, that matches mine. Interestingly, setting _march=haswell on my sandybridge produces the same cado_config.h (except for the C*FLAGS strings of course), but results in incompatible binaries.

AquilaIrreale commented on 2021-08-15 18:04

Ok, so... x86-64-v4 is clearly too high a target to run on sandybridge CPUs, but on the other hand when I tried to compile it with the lower x86-64-v2 and x86-64-v3 it just wouldn't build... neither would it if I set the various -msse4.2, -mavx etc. manually with the basic -march=x86-64. From what I read v2 is very similar in terms of capabilities to sandybridge, and v3 is to haswell.

About SIMD instructions, -march=x86-64 does have -msse2 but it seems cado wants the higher stuff. In my trials, it always failed trying to compile intrinsics that relate to the AVX instruction set. Of course as I said, manually enabling -msse4.2 and -mavx proved no solution.

It does compile (and work) with -march=sandybridge on my i7 4790K which should be a haswell, being the next major architecture right after sandybridge it would be strange otherwise... but still... it may be that sandybridge is the baseline architecture for this package, being the lowest "phisical" architecture to have AVX and x86-64-v2 not working for some reason.

Anyhow, this is the resulting cado_config.h, looking about right I think

ccorn commented on 2021-08-15 16:41

In the patch for issue #30019, if -march is detected during configuration, HAVE_* feature flags are not set at all

They are set somewhere, though the patch superficially suggests otherwise. The configuration produces a file $srcdir/cado-nfs/build/$USER/cado_config.h. If you inspect that, you will find that the HAVE_* macros are set according to your -march spec. Which is why building with -march=x86-64-v4 can be done on my sandybridge platform, except that the produced binaries cannot be run there. However, CADO-NFS seems to require some form of SIMD datatype, and the makepkg default -march=x86-64 apparently provides none, which causes build attempts to fail.

Out of curiosity, I'd like to know whether anyone of you can build with -march=sandybridge, and how this would change your cado_config.h.

AquilaIrreale commented on 2021-08-15 13:54

Alright, let's set -march to native for now then, after all, binary portability is not of primary importance here on the AUR and at least it'll work in most cases.

Still, I do not like this very much, there must be something wrong with the build system, and at the moment I don't have the time to check it thoroughly: what's the point of checking for availability of certain instruction sets such as AVX, setting flags in the form of HAVE_AVX, and then trying to use those instruction sets anyways even when compiling for platforms that do not have those instructions? In the patch for issue #30019, if -march is detected during configuration, HAVE_* feature flags are not set at all, which depending on how they are used by the build system may mean they are wrongly assumed false. In any case, I would have expected such flags to trigger some kind of conditional compilation to build a portable version of the software when specific platform features are not available, and if that's not the case at least to stop the build early since it clearly can't be completed. The fact x86-64-v4 compiles but crashes at runtime is quite baffling too.

Anyhow that's the way it is, for now at least.

@ccorn I followed the outline of your patch but implemented it a little differently to account for IFS in the word splitting of C*FLAGS.

I also removed i686 and pentium4 from the arch array, since they clearly cannot be supported for the time being.

ccorn commented on 2021-08-13 17:27

If I understand the issue correctly, previous builds were effectively -march=native because the auto-configuration examined the build machine and added SIMD support flags according to its capabilities. The recent patch now prevents this if -march is specified.

Thus, previous builds have been tuned to the build system even if that has not been explicit. So far, I have not seen a bug report about that. Therefore, I conclude that -march=native is the major use case. After all, factorization projects tend to be long-running, and not tuning the builds would result in wasting lots of time, energy, and money.

ccorn commented on 2021-08-13 16:48

For now, I found out what I think the baseline architecture to have those instructions is, so that binary packages built with this PKGBUILD can still be somewhat redistributable (although owners of older x86_64 CPUs are screwed for now).

For me, x86-64-v4 does not work: The built programs crash in the polynomial selection stage for a simple 90-digit factorization test. I tried x86-64-v3 which did not work either.

However, -march=native and -march=sandybridge do work for me. My build machine has Xeon E5-2670 which are sandybridge plus AES. It has AVX, but not AVX2.

If sandybridge does not work for you even if x86-64-v4 does, I suppose that the specified -march must be close to -march=native. Which would make -march=native the only sensible choice (and the only one consistent with more arch entries than x86_64), but then the built package will not be usable on older-generation platforms.

There is a related issue and a related patchset.

By the way, I have chosen to also set CXXFLAGS and delete other -march specs:

--- a/PKGBUILD
+++ b/PKGBUILD
@@ -22,6 +22,9 @@ provides=('cado-nfs')
 source=("git+https://gitlab.inria.fr/cado-nfs/${_pkg}.git")
 md5sums=('SKIP')

+# Need -march for SIMD support. Makes the resulting package less portable.
+_march=native
+
 pkgver() {
   cd "$_pkg"
   git log -1 --format="%cd.%h" --date=short | sed 's/-//g'
@@ -29,6 +32,13 @@ pkgver() {

 build() {
   cd "$_pkg"
+  shopt -s extglob      # for !()
+  CFLAGS+=" "
+  CFLAGS="${CFLAGS//-march=!(* *) } -march=$_march"
+  CPPFLAGS+=" "
+  CPPFLAGS="${CPPFLAGS//-march=!(* *) } -march=$_march"
+  CXXFLAGS+=" "
+  CXXFLAGS="${CXXFLAGS//-march=!(* *) } -march=$_march"
   cat <<EOF >local.sh
 PREFIX=/usr
 HWLOC=$PREFIX
@@ -37,7 +47,7 @@ CURL=$PREFIX
 # Remove 32-bit barriers to big factorizations
 FLAGS_SIZE="-DSIZEOF_P_R_VALUES=8 -DSIZEOF_INDEX=8"
 EOF
-  make CFLAGS="$CFLAGS -march=x86-64-v4" CPPFLAGS="$CPPFLAGS -march=x86-64-v4"
+  make
 }

 package() {

AquilaIrreale commented on 2021-08-13 07:38

@Dylan14 thank you for your feedback, it seems the current version of cado-nfs uses Intel intrinsics that depend on very recent versions of the AVX instruction set on x86_64.

For now, I found out what I think the baseline architecture to have those instructions is, so that binary packages built with this PKGBUILD can still be somewhat redistributable (although owners of older x86_64 CPUs are screwed for now).

When I am a little less busy I am gonna check the build system to see if there's some flag to enable more portable code to support those older architectures.

AquilaIrreale commented on 2021-08-11 14:21

@Dylan14 could you please compile the current PKGBUILD with LC_ALL=C makepkg -L and send me the resulting log file?