Athenaのパーティションを一括作成して一括削除
#!/bin/sh
# ex.
# $ ./xx.sh create 2018-01-01 2018-12-31
# $ ./xx.sh delete 2018-01-01 2018-12-31
METHOD=$1
STARTDATE=$2
ENDDATE=$3
CURRENTDATE=$STARTDATE
REGION=us-east-1
INPUTS3BUCKET=s3://xxx
OUTPUTS3BUCKET=s3://xxx
DBTABLE=データベース名.テーブル名
while [ 1 ] ; do
CURRENTYEAR=`date -j -f "%Y-%m-%d" "$CURRENTDATE" "+%Y"`
CURRENTMONTH=`date -j -f "%Y-%m-%d" "$CURRENTDATE" "+%m"`
CURRENTDAY=`date -j -f "%Y-%m-%d" "$CURRENTDATE" "+%d"`
if [ $METHOD = "create" ] ; then
COMMAND="ALTER TABLE $DBTABLE ADD PARTITION (year='$CURRENTYEAR',month='$CURRENTMONTH',day='$CURRENTDAY') location '$INPUTS3BUCKET/$CURRENTYEAR/$CURRENTMONTH/$CURRENTDAY/'"
elif [ $METHOD = "delete" ] ; then
COMMAND="ALTER TABLE $DBTABLE DROP PARTITION (year='$CURRENTYEAR',month='$CURRENTMONTH',day='$CURRENTDAY');"
else
echo please input create or delete.
exit 0
fi
aws --region $REGION athena start-query-execution --query-string "$COMMAND" --result-configuration OutputLocation=$OUTPUTS3BUCKET
if [ $CURRENTDATE = $ENDDATE ] ; then
break
fi
CURRENTDATE=`date -v+1d -j -f "%Y-%m-%d" "$CURRENTDATE" "+%Y-%m-%d"`
done
参考にさせてもらいました。
https://qiita.com/ssossan/items/6a4320b9e40af9a132c3
Author And Source
この問題について(Athenaのパーティションを一括作成して一括削除), 我々は、より多くの情報をここで見つけました https://qiita.com/hmaruyama/items/044e4c783ddf2ebf04e6著者帰属:元の著者の情報は、元のURLに含まれています。著作権は原作者に属する。
Content is automatically searched and collected through network algorithms . If there is a violation . Please contact us . We will adjust (correct author information ,or delete content ) as soon as possible .