Package Details: scylla 2.1.3-1

Git Clone URL: https://aur.archlinux.org/scylla.git (read-only)
Package Base: scylla
Description: Scylla is an open source NoSQL drop-in replacement for Apache Cassandra with fast throughput and low latency
Upstream URL: http://www.scylladb.com/
Licenses: AGPL
Conflicts: cassandra, scylla-git
Provides: scylla
Submitter: kiz
Maintainer: Argenet (dahankzter)
Last Packager: dahankzter
Votes: 5
Popularity: 0.119924
First Submitted: 2017-03-13 07:28
Last Updated: 2018-05-07 07:45

Required by (1)

Sources (7)

Pinned Comments

Argenet commented on 2018-01-05 18:07

Please note that compiling ScyllaDB is a resource-consuming operation that uses a lot of memory and puts extra load on CPU.

Latest Comments

dahankzter commented on 2018-05-07 05:45

I can't build now it complains about a git commit not existing:

==> Validating source files with md5sums...
    scylla ... Skipped
    scylla-ami ... Skipped
    scylla-swagger-ui ... Skipped
    scylla-seastar ... Skipped
    fmt ... Skipped
    dpdk ... Skipped
    c-ares ... Skipped
==> Extracting sources...
  -> Creating working copy of scylla git repo...
Cloning into 'scylla'...
done.
fatal: 'scylla-2.1.3' is not a commit and a branch 'makepkg' cannot be created from it

Argenet commented on 2018-03-05 21:55

That looks reasonable. I have also added it to the PKGBUILD file so it will use half of the CPU threads unless the CPU only has one (quite a corner case).

Glad it finally compiled well for you and thanks for your help with reporting and resolving the issues.

mgd commented on 2018-03-05 19:50

I got it compiled using

ninja -j $(($(nproc --all)/2)) ...

Argenet commented on 2018-03-04 21:44

Thanks for the follow-up. I'll include those seds for repair/repair.cc into the PKGBUILD, have missed them.

It is saddening that even 32GB RAM is not sufficient for Scylla when running build on all CPU threads :( Still, you might want to reduce it not to a single job, but at least 4 jobs to get it done faster.

Maybe it makes sense to put some value lower than nproc --all into PKGBUILD to ensure that it doesn't fail this way.

mgd commented on 2018-03-04 21:17

Hi Argenet,

Meanwhile I have done further investigations myself.

Directly issuing the offending command works w/o error. I then changed PKGBUILD and replaced ninja -jnproc --all in the build() section by ninja -j 1 (nproc --all yields 8 on my system).

I also had to add another pair of sed work arounds: sed -i -e 's@sizeof(byte)@sizeof(CryptoPP::byte)@g' repair/repair.cc sed -i -e 's@reinterpret_cast<const byte@reinterpret_cast<const CryptoPP::byte@g' repair/repair.cc

The thus adjusted PKGBUILD seems to work but is still running aka takes a small eternity to complete.

W/r to memory: My system has 32GB of RAM which according to your mail should be fine with 8 processes per 2GB.

Argenet commented on 2018-03-04 20:27

Hi Michael,

The next failure indicates an internal compiler error. With Scylla, this typically happens when compiler runs out of memory. Scylla is using C++ templates heavily and thus it consumes a lot of RAM on compilation. You can try to decrease the number of concurrent jobs. Currently, PKGBUILD uses nproc --all to pass in the number of jobs, you can find out what its value is on your machine and edit PKGBUILD. For example, if on you machine nproc --all produces 4, you can try to make it: ninja -j2 build/release/scylla in the PKGBUILD file and see how it goes.

You can approximately expect 2GB of RAM per job to be used.

mgd commented on 2018-03-04 12:55

Hi Argenet,

your work around gets rid of the problem reported.

However now I stumble over a gcc crash later down the compile process. Need to further investigate though.

[97/297] CXX build/release/main.o FAILED: build/release/main.o g++ -MD -MT build/release/main.o -MF build/release/main.o.d -std=gnu++1y -g -Wall -Werror -Wno-error=deprecated-declarations -fvisibility=hidden -Wno-attributes -pthread -I/home/mgd/packages/scylla-2.1.0-2/src/scylla/seastar -U_FORTIFY_SOURCE -march=nehalem -I/home/mgd/packages/scylla-2.1.0-2/src/scylla/seastar/fmt -DBOOST_TEST_DYN_LINK -Wno-overloaded-virtual -Wno-maybe-uninitialized -Wno-sign-compare -Wno-error=cpp -DFMT_HEADER_ONLY -DHAVE_HWLOC -DHAVE_NUMA -DHAVE_LZ4_COMPRESS_DEFAULT -DHAVE_ASAN_FIBER_SUPPORT -O2 -I/home/mgd/packages/scylla-2.1.0-2/src/scylla/seastar/build/release/gen -I/home/mgd/packages/scylla-2.1.0-2/src/scylla/seastar/build/release/c-ares -DBOOST_TEST_DYN_LINK -Wno-maybe-uninitialized -Wno-tautological-compare -Wno-missing-braces -Wno-misleading-indentation -Wno-overflow -Wno-noexcept-type -Wno-nonnull-compare -Wno-error=deprecated-declarations -DHAVE_LIBSYSTEMD=1 -I. -I build/release/gen -I seastar -I seastar/build/release/gen -c -o build/release/main.o main.cc g++: interner Compiler-Fehler: Getötet (Programm cc1plus) Bitte senden Sie einen vollständigen Fehlerbericht auf Englisch ein; inclusive vorverarbeitetem Quellcode, wenn es dienlich ist. Weitere Hinweise finden Sie unter »<https://bugs.archlinux.org/>«.

Argenet commented on 2018-03-04 00:28

The problem occurs because of the breaking change introduced in Crypto++ that hides the byte typedef from the global namespace into the CryptoPP:: namespace.

I have opened a GitHub issue for that: https://github.com/scylladb/scylla/issues/3252

The problem is worked around with sed scripts in PKGBUILD file for now. Please try again, it should compile fine.

Argenet commented on 2018-03-03 19:30

Hi Michael, thanks for reporting this issue. I was able to reproduce it, so far it seems to be caused by some changes in the recently updated Crypto++ library. Let me see if some fix or workaround can be applied to resolve this.

mgd commented on 2018-03-03 16:47

Building scylla 2.1.0 fails for me. It seems the type byte is not properly defined for unknown reasons. My system is up-to-date AFAICT.

Find below an excerpt from the build log: [42/297] ANTLR3 cql3/Cql.g [43/297] CXX build/release/gen/cassandra_types.o [44/297] CXX build/release/gen/cql3/CqlLexer.o [45/297] CXX build/release/gen/cql3/CqlParser.o FAILED: build/release/gen/cql3/CqlParser.o g++ -MD -MT build/release/gen/cql3/CqlParser.o -MF build/release/gen/cql3/CqlParser.o.d -std=gnu++1y -g -Wall -Werror -Wno-error=deprecated-declarations -fvisibility=hidden -Wno-attributes -pthread -I/home/mgd/packages/scylla-2.1.0-1/src/scylla/seastar -U_FORTIFY_SOURCE -march=nehalem -I/home/mgd/packages/scylla-2.1.0-1/src/scylla/seastar/fmt -DBOOST_TEST_DYN_LINK -Wno-overloaded-virtual -Wno-maybe-uninitialized -Wno-sign-compare -Wno-error=cpp -DFMT_HEADER_ONLY -DHAVE_HWLOC -DHAVE_NUMA -DHAVE_LZ4_COMPRESS_DEFAULT -DHAVE_ASAN_FIBER_SUPPORT -O2 -I/home/mgd/packages/scylla-2.1.0-1/src/scylla/seastar/build/release/gen -I/home/mgd/packages/scylla-2.1.0-1/src/scylla/seastar/build/release/c-ares -DBOOST_TEST_DYN_LINK -Wno-maybe-uninitialized -Wno-tautological-compare -Wno-missing-braces -Wno-misleading-indentation -Wno-overflow -Wno-noexcept-type -Wno-nonnull-compare -Wno-error=deprecated-declarations -DHAVE_LIBSYSTEMD=1 -I. -I build/release/gen -I seastar -I seastar/build/release/gen -fno-sanitize-address-use-after-scope -c -o build/release/gen/cql3/CqlParser.o build/release/gen/cql3/CqlParser.cpp In file included from ./query-result.hh:28:0, from ./mutation_partition.hh:38, from ./mutation.hh:26, from ./database.hh:62, from ./service/storage_proxy.hh:44, from ./db/schema_tables.hh:43, from ./db/system_keyspace.hh:46, from ./cql3/functions/function_name.hh:45, from ./cql3/selection/selectable.hh:49, from ./cql3/selection/writetime_or_ttl.hh:45, from build/release/gen/cql3/CqlParser.hpp:63, from build/release/gen/cql3/CqlParser.cpp:45: ./md5_hasher.hh: In Elementfunktion »void md5_hasher::update(const char*, size_t)«: ./md5_hasher.hh:34:46: Fehler: »byte« wurde in diesem Gültigkeitsbereich nicht definiert static_assert(sizeof(char) == sizeof(byte), "Assuming lengths will be the same"); ^~~~ ./md5_hasher.hh:34:46: Anmerkung: empfohlene Alternative: In file included from /usr/include/cryptopp/cryptlib.h:102:0, from /usr/include/cryptopp/iterhash.h:6, from /usr/include/cryptopp/md5.h:4, from ./query-result.hh:25, from ./mutation_partition.hh:38, from ./mutation.hh:26, from ./database.hh:62, from ./service/storage_proxy.hh:44, from ./db/schema_tables.hh:43, from ./db/system_keyspace.hh:46, from ./cql3/functions/function_name.hh:45, from ./cql3/selection/selectable.hh:49, from ./cql3/selection/writetime_or_ttl.hh:45, from build/release/gen/cql3/CqlParser.hpp:63, from build/release/gen/cql3/CqlParser.cpp:45: /usr/include/cryptopp/config.h:228:23: Anmerkung: »CryptoPP::byte« typedef unsigned char byte; ^~~~ In file included from ./query-result.hh:28:0, from ./mutation_partition.hh:38, from ./mutation.hh:26, from ./database.hh:62, from ./service/storage_proxy.hh:44, from ./db/schema_tables.hh:43, from ./db/system_keyspace.hh:46, from ./cql3/functions/function_name.hh:45, from ./cql3/selection/selectable.hh:49, from ./cql3/selection/writetime_or_ttl.hh:45, from build/release/gen/cql3/CqlParser.hpp:63, from build/release/gen/cql3/CqlParser.cpp:45: ./md5_hasher.hh:35:44: Fehler: ISO-C++ verbietet Deklaration von »type name« ohne Typ [-fpermissive] hash.Update(reinterpret_cast<const byte*="">(ptr), length * sizeof(byte)); ^~~~ ./md5_hasher.hh:35:44: Fehler: expected »>« before »byte« ./md5_hasher.hh:35:44: Fehler: expected »(« before »byte« ./md5_hasher.hh:35:49: Fehler: expected primary-expression before »>« token hash.Update(reinterpret_cast<const byte*="">(ptr), length * sizeof(byte)); ^</const></const>

All comments