Package Details: apache-spark 2.4.0-1

Git Clone URL: (read-only)
Package Base: apache-spark
Description: fast and general engine for large-scale data processing
Upstream URL:
Licenses: Apache
Submitter: huitseeker
Maintainer: lukaszimmermann
Last Packager: huitseeker
Votes: 40
Popularity: 0.060241
First Submitted: 2015-10-04 09:31
Last Updated: 2019-01-30 19:46

Dependencies (9)

Required by (0)

Sources (7)

Latest Comments

« First ‹ Previous 1 2 3 4 5 Next › Last »

Knight commented on 2016-03-12 10:25

@huitseeker, sorry for the delay. I don't know why but it's weird that the installation succeeded this time...

huitseeker commented on 2016-03-09 08:11

Hi @Knight,

The exact nature of the error is above the maven output you paste here. without it, it's difficult to interpret this message.

Knight commented on 2016-03-09 07:12

[INFO] Spark Project Parent POM ........................... SUCCESS [04:12 min]
[INFO] Spark Project Test Tags ............................ SUCCESS [ 58.838 s]
[INFO] Spark Project Launcher ............................. SUCCESS [01:34 min]
[INFO] Spark Project Networking ........................... SUCCESS [ 31.249 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 36.530 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [01:00 min]
[INFO] Spark Project Core ................................. FAILURE [04:08 min]


[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:05 min
[INFO] Finished at: 2016-03-09T15:04:30+08:00
[INFO] Final Memory: 59M/735M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-core_2.11: Exe
cution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed -> [Help 1]

what should I do?

Aetf commented on 2016-02-23 04:43

The the package function should be

@@ -55,7 +55,7 @@ package() {

mkdir -p $pkgdir/etc/profile.d
echo '#!/bin/sh' > $pkgdir/etc/profile.d/
- echo 'SPARK_HOME=$pkgdir/usr/share/apache-spark' >> $pkgdir/etc/profile.d/
+ echo 'SPARK_HOME=/usr/share/apache-spark' >> $pkgdir/etc/profile.d/
echo 'export SPARK_HOME' >> $pkgdir/etc/profile.d/
chmod 755 $pkgdir/etc/profile.d/

aftabnack commented on 2016-02-17 07:35

My compiling is failed at the point of building MQTT, every time its running out of memory. I'm running arch 32-bit in virtual box with 5GB RAM.
It failed with outOfMemory error with message saying "malloc failed to allocate 171128 bytes for chunk::new"

I have run it about 8times.

I tried modifying PKGBUILD and editing MAVEN_OPTS with different values for
-Xmx MaxPermSize. I can share the hs_err/replay file.

oneeyed commented on 2016-01-07 18:38

Adding two files in /etc/profile.d (sh and csh) to set SPARK_HOME would definitely help users of this package.

StefanK2 commented on 2016-01-05 14:46

LOL I found the problem, SPARK_HOME wasn't set ;-)
Everything is working now.
I just updated the package, but now I get this error-message when I call spark-submit or spark-shell:

ls: cannot access /usr/assembly/target/scala-2.10: No such file or directory
Failed to find Spark assembly in /usr/assembly/target/scala-2.10.
You need to build Spark before running this program.

According to pacman scala 2.11 is installed on my machine.
Also when I checked in apache-spark/src/spark-1.6.0/assembly/target there is only a folder scala-2.11.

Did I miss some configuration-step or something?
I changed the line 36 in PKGBUILD from
dev/ 2.11

to get rid of the deprecation warning (I assumed that my problem is related to the warning, but it didn't change it).

Also I just found this: seems to be a spark problem.

huitseeker commented on 2015-12-21 14:44

@fosskers I really can't reproduce your issue. Do you have more info on your config ? Have you thought of clearing your maven cache before rebuilding ?

flying-sheep commented on 2015-11-23 14:16

yes, do this please.

1. it’s compiled from git master, so it should have a -git suffix after the name
3. the version is wrong (1.6.0 isn’t released yet, you’ll have to use some version generated by a pkgver() function)
2. the underscore is against convention (except if the name literally has an underscore in it)

fosskers commented on 2015-11-19 20:59

Have spent the last few hours trying to compile this. At the point of building `spark-streaming-mqtt`I always either run out of RAM (I have 4G on my laptop) or I see this:

[INFO] --- maven-assembly-plugin:2.5.5:single (test-jar-with-dependencies) @ spark-streaming-mqtt_2.11 ---
[INFO] Reading assembly descriptor: src/main/assembly/assembly.xml
[WARNING] The assembly descriptor contains a filesystem-root relative reference, which is not cross platform compatible /
[INFO] Building jar: /home/colin/building/apache-spark/src/spark-1.5.2/external/mqtt/target/scala-2.11/spark-streaming-mqtt-test-1.5.2.jar
/home/colin/building/apache-spark/src/spark-1.5.2/build/mvn: line 152: 28401 Killed ${MVN_BIN} -DzincPort=${ZINC_PORT} "$@"

Any ideas?