diff options
author | François Garillot | 2016-07-12 00:32:25 +0200 |
---|---|---|
committer | François Garillot | 2016-07-12 00:42:23 +0200 |
commit | 6c2bfc0675639904047b6fbb38789f5c0c569823 (patch) | |
tree | 36c63a87cbc0766d5de4a4dc30acc9efd5d45234 /spark-daemon-run.sh | |
parent | 59c75dba2625d4dff23108983d7396c189355a62 (diff) | |
download | aur-6c2bfc0675639904047b6fbb38789f5c0c569823.tar.gz |
An update from Xiang Gao <qasdfgtyuiop@gmail.com>
Squashed commit of the following:
commit 1700b03fcd6b95198564e07345c482dbc85c7c92
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Sun Jul 3 08:39:50 2016 -0400
add support for slave service
commit 70aecf80a40341836dfa3b1cae222b1db74801f3
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Sun Jul 3 08:12:07 2016 -0400
fix usage
commit f810e7d98b67732af1814a36a65843f1a9b0fae9
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Sun Jul 3 08:05:57 2016 -0400
add service for master
commit 0f597e790c48b3487f47508d1eac785b45248bb0
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Sun Jul 3 04:32:16 2016 -0400
add scripts to run spark in foreground
commit 4eace18e92513bfa5f3fc8fe91a020f55fd67800
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Fri Jul 1 05:32:16 2016 -0400
add rsync as opt deps
commit a5ea131071667416b14cf95ffbe18bef4e9c968f
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Wed Jun 29 22:44:05 2016 -0400
add hadoop as dependency
commit bde492e3354cb2164855f79c49c42037c14d42a4
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Wed Jun 29 22:22:26 2016 -0400
change log files to /var/log/apache-spark
fix sparkenv.sh to load hadoop classpaths
commit 4625ceb7b05832217a4e1a1c46fe16f645d30fc4
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Wed Jun 29 07:20:35 2016 -0400
fix systemd service file
commit 2354f598de49424942804051f62342b42b3eb1f3
Author: Xiang Gao <qasdfgtyuiop@gmail.com>
Date: Wed Jun 29 07:04:03 2016 -0400
Make a lot of changes:
* upgrade to 1.6.2
* use pre-built binaries "spark--bin-without-hadoop.tgz" instead of compiling from sources
* remove dependency on scala and maven which is already built in spark
* move python2 and hadoop from depends to optdepends
* add r as optdepends
* move templates for conf to conf-templates
* move the whole conf directory to /etc/apache-spark
* move apache-spark directory to /opt
* move the work directory to /var/lib/apache-spark
Diffstat (limited to 'spark-daemon-run.sh')
-rwxr-xr-x | spark-daemon-run.sh | 139 |
1 files changed, 139 insertions, 0 deletions
diff --git a/spark-daemon-run.sh b/spark-daemon-run.sh new file mode 100755 index 000000000000..34e3a80fa37a --- /dev/null +++ b/spark-daemon-run.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Runs a Spark daemon foreground. +# +# Environment Variables +# +# SPARK_CONF_DIR Alternate conf dir. Default is ${SPARK_HOME}/conf. +# SPARK_LOG_DIR Where log files are stored. ${SPARK_HOME}/logs by default. +# SPARK_MASTER host:path where spark code should be rsync'd from +# SPARK_IDENT_STRING A string representing this instance of spark. $USER by default +# SPARK_NICENESS The scheduling priority for daemons. Defaults to 0. +## + +usage="Usage: spark-daemon-run.sh [--config <conf-dir>] (class|submit) <spark-command> <spark-instance-number> <args...>" + +# if no args specified, show usage +if [ $# -le 1 ]; then + echo $usage + exit 1 +fi + +if [ -z "${SPARK_HOME}" ]; then + export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)" +fi + +. "${SPARK_HOME}/sbin/spark-config.sh" + +# get arguments + +# Check if --config is passed as an argument. It is an optional parameter. +# Exit if the argument is not a directory. + +if [ "$1" == "--config" ] +then + shift + conf_dir="$1" + if [ ! -d "$conf_dir" ] + then + echo "ERROR : $conf_dir is not a directory" + echo $usage + exit 1 + else + export SPARK_CONF_DIR="$conf_dir" + fi + shift +fi + +mode=$1 +shift +command=$1 +shift +instance=$1 +shift + +spark_rotate_log () +{ + log=$1; + num=5; + if [ -n "$2" ]; then + num=$2 + fi + if [ -f "$log" ]; then # rotate logs + while [ $num -gt 1 ]; do + prev=`expr $num - 1` + [ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num" + num=$prev + done + mv "$log" "$log.$num"; + fi +} + +. "${SPARK_HOME}/bin/load-spark-env.sh" + +if [ "$SPARK_IDENT_STRING" = "" ]; then + export SPARK_IDENT_STRING="$USER" +fi + + + +# get log directory +if [ "$SPARK_LOG_DIR" = "" ]; then + export SPARK_LOG_DIR="${SPARK_HOME}/logs" +fi +mkdir -p "$SPARK_LOG_DIR" +touch "$SPARK_LOG_DIR"/.spark_test > /dev/null 2>&1 +TEST_LOG_DIR=$? +if [ "${TEST_LOG_DIR}" = "0" ]; then + rm -f "$SPARK_LOG_DIR"/.spark_test +else + chown "$SPARK_IDENT_STRING" "$SPARK_LOG_DIR" +fi + +# some variables +log="$SPARK_LOG_DIR/spark-$SPARK_IDENT_STRING-$command-$instance-$HOSTNAME.out" + +# Set default scheduling priority +if [ "$SPARK_NICENESS" = "" ]; then + export SPARK_NICENESS=0 +fi + +if [ "$SPARK_MASTER" != "" ]; then + echo rsync from "$SPARK_MASTER" + rsync -a -e ssh --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' "$SPARK_MASTER/" "${SPARK_HOME}" +fi + +spark_rotate_log "$log" +echo "running $command, logging to $log" + +case "$mode" in + (start) + exec nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-class $command "$@" >> "$log" 2>&1 < /dev/null + ;; + + (submit) + exec nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-submit --class $command "$@" >> "$log" 2>&1 < /dev/null + ;; + + (*) + echo "unknown mode: $mode" + exit 1 + ;; +esac |