Sparkの古典的なケースの非構造データ処理

需要:tomcatログに基づいてurlアクセス状況を計算し、具体的なurlは以下のように要求する:統計GETとPOSTのURLアクセス量を区別した結果、アクセス方式、URL、アクセス量テストデータセット:CODEでコードシートを見て私のコードシート1966.168.2.1--[03/Jul/2014 4:23:36:38+0800]「GET/course/detail/3.html HTTP/1.0」200 38435 0.038182.131.89.195--[03/Jul/2014:23:37:43+0800」「GET/html/notes/20140617/888.html HTTP/1.0」301-0.0001196.168.2.1--[03/Jul/2014:23:38:38:38:27+0800]「POST/service/notes/adVidVidVidVidVidVidVidVidVidVidVidVidVid/POT/POT/POT/S ewTimes_23.html HTTP/1.0」 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:39:03 +0800] “GET/html/notes/20140617/779.html HTTP/1.0” 200 69539 0.046 196.168.2.1 - - [03/Jul/2014:23:43:00 +0800] “GET/html/notes/20140318/24.html HTTP/1.0” 200 67171 0.049 196.168.2.1 - - [03/Jul/2014:23:43:59 +0800] “POST/service/notes/addViewTimes_779.htm HTTP/1.0” 200 1 0.003 196.168.2.1 - - [03/Jul/2014:23:45:51 +0800] “GET/html/notes/20140617/888.html HTTP/1.0” 200 70044 0.060 196.168.2.1 - - [03/Jul/2014:23:46:17 +0800] “GET/course/list/73.htm HTTP/1.0” 200 12125 0.010 196.168.2.1 - - [03/Jul/2014:23:46:58 +0800] “GET/html/notes/20140609/542.html HTTP/1.0” 200 94971 0.077 196.168.2.1 - - [03/Jul/2014:23:48:31 +0800] “POST/service/notes/addViewTimes_24.htm HTTP/1.0” 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:48:34 +0800] “POST/service/notes/addViewTimes_542.htm HTTP/1.0” 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:49:31 +0800] “GET/notes/index-top-3.htm HTTP/1.0” 200 53494 0.041 196.168.2.1 - - [03/Jul/2014:23:50:55 +0800] “GET/html/notes/20140609/544.html HTTP/1.0” 200 183694 0.076 196.168.2.1 - - [03/Jul/2014:23:53:32 +0800] “POST/service/notes/addViewTimes_544.htm HTTP/1.0” 200 2 0.004 196.168.2.1 - - [03/Jul/2014:23:54:53 +0800] “GET/service/notes/addViewTimes_900.htm HTTP/1.0” 200 151770 0.054 196.168.2.1 - - [03/Jul/2014:23:57:42 +0800] “GET/html/notes/20140620/872.html HTTP/1.0” 200 52373 0.034 196.168.2.1 - - [03/Jul/2014:23:58:17 +0800] “POST/service/notes/addViewTimes_900.htm HTTP/1.0” 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:58:51 +0800] “GET/html/notes/20140617/888.html HTTP/1.0” 200 70044 0.057 186.76.76.76 - - [03/Jul/2014:23:48:34 +0800] “POST/service/notes/addViewTimes_542.htm HTTP/1.0” 200 2 0.003 186.76.76.76 - - [03/Jul/2014:23:46:17 +0800] “GET/course/list/73.htm HTTP/1.0” 200 12125 0.010 8.8.8.8 - - [03/Jul/2014:23:46:58 +0800] “GET/html/notes/20140609/542.html HTTP/1.0” 200 94971 0.077
Tomcatログは不規則なため、洗浄データをフィルタリングする必要があります.

package ClassicCase

import org.apache.spark.{SparkConf, SparkContext}

/**
  *  ： 
  * Created by YJ on 2017/2/8.
  */


object case7 {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local").setAppName("reduce")
    val sc = new SparkContext(conf)
    sc.setLogLevel("ERROR")
    val data = sc.textFile("hdfs://192.168.109.130:8020//user/flume/ClassicCase/case7/*")

    //filter  0，  GET POST URL   
    val filtered = data.filter(_.length() > 0).filter(line => (line.indexOf("GET") > 0 || line.indexOf("POST") > 0))

    //   
    val res = filtered.map(line => {
      if (line.indexOf("GET") > 0) {
        //  GET  URL   
        (line.substring(line.indexOf("GET"), line.indexOf("HTTP/1.0")).trim, 1)
      } else {
        //  POST  URL   
        (line.substring(line.indexOf("POST"), line.indexOf("HTTP/1.0")).trim, 1)
      } // reduceByKey sum  
    }).reduceByKey(_ + _)

    // action   
    res.collect()
  }
}

出力結果(POST/service/notes/addViewTimes_779.html,1),(GET/service/notes/addViewTimes_900.html,1),(POST/service/notes/addViewTimes_900.html,1),(GET/notes/index-top-3.html,1),(GET/notes/notes/index/index-top-3.html,(GET/html/notes/201440318/248/248.html,1),(GET/html/notes/2014609/544.html,1),(POST/service/service/svice/notes/add/addnotes/addViewTimes_542.htm,2),(POST/service/notes/addViewTimes_544.htm,1),(GET/html/notes/20140609/542.html, (POST/service/notes/addViewTimes_23.htm,1), (GET/html/notes/20140617/888.html,3), (POST/service/notes/addViewTimes_24.htm,1), (GET/course/detail/3.htm,1), (GET/course/list/73.htm,2), (GET/html/notes/20140617/779.html,1), (GET/html/notes/20140620/872.html,1)