AWS lambdaでS3オブジェクトの自動ウイルススキャンをやろうとしてハマった話

7458 ワード
lambda AWS Python3 S3 Python Python テキストリンク

やりたかったこと

S3にファイルがアップされたら自動でウイルススキャンを行いたい
調べてみると以下がいい感じだったので実施したが思いの外ハマったのでメモ
https://github.com/upsidetravel/bucket-antivirus-function

手順

権限周りは以下を参考にしてください
https://dev.classmethod.jp/articles/s3-bucket-antivirus-lambda/

git clone https://github.com/upsidetravel/bucket-antivirus-function.git
cd bucket-antivirus-function/
sudo make all

./build/lambda.zipが作成されるのでS3にアップする

以下のCfnテンプレートで３時間ごとに実行されるlambda関数作成

Resources:
  ScheduledRule:
    Type: AWS::Events::Rule
    Properties:
      ScheduleExpression: 'cron(00 0-23/3 * * ? *)'
      State: ENABLED
      Targets:
        - Arn: !GetAtt Lambda.Arn
          Id: bucket-antivirus-update
  PermissionForEventsToInvokeLambda:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref Lambda
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt ScheduledRule.Arn
  Lambda:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: lambda-function-bucket
        S3Key: lambda.zip
      Environment:
        Variables:
          AV_DEFINITION_S3_BUCKET: antivirus-definition-bucket
      FunctionName: bucket-antivirus-update
      Handler: update.lambda_handler
      MemorySize: 1024
      Role: antivirus-update-role-arn
      Runtime: python3.7
      Timeout: 300

エラー発生

とりあえずここまでの工程でS3にウイルス定義ファイルをアップする部分はできたはずなので試しに実行してみると以下のエラー発生

START RequestId: xxxxxxxxxxxxxxxxxxxxxxxx Version: $LATEST
Script starting at 2021/04/19 04:37:41 UTC

Attempting to create directory /tmp/clamav_defs.

Starting freshclam with defs in /tmp/clamav_defs.
freshclam output:
b'./bin/freshclam: error while loading shared libraries: libprelude.so.28: cannot open shared object file: No such file or directory\n'
Unexpected exit code from freshclam: 127.
File does not exist: main.cld
File does not exist: main.cvd
File does not exist: daily.cld
File does not exist: daily.cvd
File does not exist: bytecode.cld
File does not exist: bytecode.cvd
Script finished at 2021/04/19 04:37:42 UTC

END RequestId: xxxxxxxxxxxxxxxxxxxxxxxx
REPORT RequestId: xxxxxxxxxxxxxxxxxxxxxxxx    Duration: 2604.36 ms    Billed Duration: 2605 ms    Memory Size: 128 MB    Max Memory Used: 85 MB    Init Duration: 334.40 ms

あれこれ調べてみると
https://github.com/upsidetravel/bucket-antivirus-function/issues/125
こちらのissueに遭遇。どうやらlambda側がライブラリを見つけられないことが原因のよう
Dockerfileを以下のように書き換えてmakeし直したところエラーが解消しました。

FROM amazonlinux:2

# Set up working directories
RUN mkdir -p /opt/app
RUN mkdir -p /opt/app/build
RUN mkdir -p /opt/app/bin/

# Copy in the lambda source
WORKDIR /opt/app
COPY ./*.py /opt/app/
COPY requirements.txt /opt/app/requirements.txt

# Install packages
RUN yum update -y
RUN yum install -y cpio python3-pip yum-utils zip unzip less
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

# This had --no-cache-dir, tracing through multiple tickets led to a problem in wheel
RUN pip3 install -r requirements.txt
RUN rm -rf /root/.cache/pip

# Download libraries we need to run in lambda
WORKDIR /tmp
RUN yumdownloader -x \*i686 --archlist=x86_64 clamav clamav-lib clamav-update json-c pcre2 libprelude gnutls libtasn1 lib64nettle nettle
RUN rpm2cpio clamav-0*.rpm | cpio -idmv
RUN rpm2cpio clamav-lib*.rpm | cpio -idmv
RUN rpm2cpio clamav-update*.rpm | cpio -idmv
RUN rpm2cpio json-c*.rpm | cpio -idmv
RUN rpm2cpio pcre*.rpm | cpio -idmv
RUN rpm2cpio gnutls* | cpio -idmv
RUN rpm2cpio nettle* | cpio -idmv
RUN rpm2cpio lib* | cpio -idmv
RUN rpm2cpio *.rpm | cpio -idmv
RUN rpm2cpio libtasn1* | cpio -idmv

# Copy over the binaries and libraries
RUN cp /tmp/usr/bin/clamscan /tmp/usr/bin/freshclam /tmp/usr/lib64/* /opt/app/bin/

# Fix the freshclam.conf settings
RUN echo "DatabaseMirror database.clamav.net" > /opt/app/bin/freshclam.conf
RUN echo "CompressLocalDatabase yes" >> /opt/app/bin/freshclam.conf

# Create the zip file
WORKDIR /opt/app
RUN zip -r9 --exclude="*test*" /opt/app/build/lambda.zip *.py bin

WORKDIR /usr/local/lib/python3.7/site-packages
RUN zip -r9 /opt/app/build/lambda.zip *

WORKDIR /opt/app
前処理コマンド——条件コンパイル
毎日アルゴリズム(四)問題例題の検索