Cosmos DB に大量にデータを入れてみる

7006 ワード
CosmosDB Java Azure Java テキストリンク

ちょっとした検証目的のため、Cosmos DBに大量データを投じる簡易的なサンプルJavaコードを作成してみました。
当然ながら大量のトランザクションが走ると課金額も増加するため、ご利用は自己責任にてお願いします。

使用方法

1) ニューヨーク市の Taxi and Limousine Commission のページより、Yellowタクシーの乗降データをダウンロードし、任意のフォルダーに配置します。（例：C:\\Temp\\）
2) Eclipseで pom.xml をロードし、Javaパースペクティブを開きます。
3) appsettings.properties を開き、以下のプロパティー値をご自身のCosmos DB環境にあわせて更新します。CsvFileDirには、手順 1) で CSVファイルを配置したフォルダーを指定します。

EndPointUrl=https://REPLACE_ME_WITH_YOURS.documents.azure.com:443/
PrimaryKey=REPLACE_ME_WITH_YOURS
Database=TestDB
Collection=TestCollection
CsvFileDir=C:\\Temp\\

4) GenerateNYCTaxiData.java をJava Applicationとして実行します。

実行例
Created a new database:
{"_rid":"DewXAA==","id":"TestDB","_self":"dbs/DewXAA==/","_colls":"colls/","_users":"users/","_etag":"\"00002908-0000-2300-0000-5e4cf35e0000\"","_ts":1582101343}
Press any key to continue..

Created a new collection:
{"uniqueKeyPolicy":{"uniqueKeys":[]},"geospatialConfig":{"type":"Geography"},"_rid":"DewXAIlOilc=","_sprocs":"sprocs/","_triggers":"triggers/","_udfs":"udfs/","conflictResolutionPolicy":{"mode":"LastWriterWins","conflictResolutionProcedure":"","conflictResolutionPath":"/_ts"},"_docs":"docs/","partitionKey":{"paths":["/_partitionKey"],"kind":"Hash"},"indexingPolicy":{"indexingMode":"consistent","automatic":true,"includedPaths":[{"path":"/*"}],"excludedPaths":[{"path":"/\"_etag\"/?"}]},"id":"TestCollection","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/","_conflicts":"conflicts/","_etag":"\"00001f05-0000-2300-0000-5e4cf3620000\"","_ts":1582101346}
Press any key to continue..
Read a csv file:
C:\Temp\yellow_tripdata_2019-01.csv
Press any key to continue..

Header: VendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,RatecodeID,store_and_fwd_flag,PULocationID,DOLocationID,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount,congestion_surcharge
Created 1 document:
{"_rid":"DewXAIlOilcBAAAAAAAAAA==","puLocationid":"151","fare_amount":"7","tpep_dropoff_datetime":"2019-01-01 00:53:20","congestion_surcharge":"","vendorid":"1","passenger_count":"1","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":"1.50","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"1","ratecodeid":"1","total_amount":"9.95","extra":"0.5","tip_amount":"1.65","mta_tax":"0.5","id":"92ce47d1-b876-443e-8e47-d1b876643e4a","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcBAAAAAAAAAA==/","doLocationid":"239","tpep_pickup_datetime":"2019-01-01 00:46:40","_etag":"\"00003108-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 2 document:
{"_rid":"DewXAIlOilcCAAAAAAAAAA==","puLocationid":"239","fare_amount":"14","tpep_dropoff_datetime":"2019-01-01 01:18:59","congestion_surcharge":"","vendorid":"1","passenger_count":"1","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":"2.60","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"1","ratecodeid":"1","total_amount":"16.3","extra":"0.5","tip_amount":"1","mta_tax":"0.5","id":"14a612b4-4eac-4aa3-a612-b44eacdaa3b6","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcCAAAAAAAAAA==/","doLocationid":"246","tpep_pickup_datetime":"2019-01-01 00:59:47","_etag":"\"00003208-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 3 document:
{"_rid":"DewXAIlOilcDAAAAAAAAAA==","puLocationid":"236","fare_amount":"4.5","tpep_dropoff_datetime":"2018-12-21 13:52:40","congestion_surcharge":"","vendorid":"2","passenger_count":"3","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":".00","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"1","ratecodeid":"1","total_amount":"5.8","extra":"0.5","tip_amount":"0","mta_tax":"0.5","id":"69d7e305-53e1-4efd-97e3-0553e18efd36","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcDAAAAAAAAAA==/","doLocationid":"236","tpep_pickup_datetime":"2018-12-21 13:48:30","_etag":"\"00003308-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 4 document:
{"_rid":"DewXAIlOilcEAAAAAAAAAA==","puLocationid":"193","fare_amount":"3.5","tpep_dropoff_datetime":"2018-11-28 15:55:45","congestion_surcharge":"","vendorid":"2","passenger_count":"5","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":".00","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"2","ratecodeid":"1","total_amount":"7.55","extra":"0.5","tip_amount":"0","mta_tax":"0.5","id":"3cb3e88a-da63-4060-b3e8-8ada63d060b3","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcEAAAAAAAAAA==/","doLocationid":"193","tpep_pickup_datetime":"2018-11-28 15:52:25","_etag":"\"00003408-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 5 document:
{"_rid":"DewXAIlOilcFAAAAAAAAAA==","puLocationid":"193","fare_amount":"52","tpep_dropoff_datetime":"2018-11-28 15:58:33","congestion_surcharge":"","vendorid":"2","passenger_count":"5","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":".00","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"2","ratecodeid":"2","total_amount":"55.55","extra":"0","tip_amount":"0","mta_tax":"0.5","id":"ff1002e6-03f5-4bac-9002-e603f5cbacf5","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcFAAAAAAAAAA==/","doLocationid":"193","tpep_pickup_datetime":"2018-11-28 15:56:57","_etag":"\"00003508-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 6 document:
{"_rid":"DewXAIlOilcGAAAAAAAAAA==","puLocationid":"193","fare_amount":"3.5","tpep_dropoff_datetime":"2018-11-28 16:28:26","congestion_surcharge":"","vendorid":"2","passenger_count":"5","tolls_amount":"5.76","improvement_surcharge":"0.3","trip_distance":".00","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"2","ratecodeid":"1","total_amount":"13.31","extra":"0.5","tip_amount":"0","mta_tax":"0.5","id":"3cf17f8b-617e-49a4-b17f-8b617e29a49c","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcGAAAAAAAAAA==/","doLocationid":"193","tpep_pickup_datetime":"2018-11-28 16:25:49","_etag":"\"00003608-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 7 document:

:

参考情報