【GCP】Professional Cloud Architect - Case Study


MountKirk Games

Background

  • a new game expect to be very polular.
  • Plan to deploy backend to Compute Engine.
  • Capture streaming metric
  • Run intensive analytics
    • Cloud Data Loss Prevention API
  • integrate with a managed NoSQL database
    • Migrate MySQL to BigTable or Datastore

Business requirements

  • Increase to a global footprint
    • Use a global HTTP load balancer that serves managed instances groups that have autoscaling enabled
  • Improve uptime-downtime
  • Increase efficiency of the cloud resources
    • Migrate MySQL to BigTable or Datastore
  • Reduce latency to all customers

Technical requirements

  • Backend platform
    • Dynamically scale up or down based on game activity
      • Managed Instance Group
      • Use Kubernetes Engine if they want create microservices.
    • Connect to a transactional database service to manage user profiles and game state
      • Cloud Spanner
    • Store game activity in a timeseries database service for future analysis
      • BigTable for store datas
      • BigQuery for analysis
    • No data lost due to processing backlogs
      • Cloud Pub/Sub
    • Custom Linux distro
      • Cannot use App Engine. (Standard or flexable)
      • Compute Engine Only
      • Use a global HTTP load balancer that serves managed instances groups that have autoscaling enabled
  • Analytics platform
    • Realtime Analytics
      • Streaming Pipeline: Cloud Pub/Sub -> Cloud Dataflow -> BigQuery
      • Batch Pipeline: Cloud Storage -> Cloud Dataflow -> BigQuery
    • Dynamically scale up or down based on game activity
      • Use BigQuery for analytics
    • Process incoming data on the fly directly from the game servers
      • Use Cloud Pub/Sub handles streaming data in real time
    • Process data that arrives late because of slow mobile networks
    • Allow queries to access at least 10 TB of historical data
      • Big Query
    • Process files uploaded by user's mobile

Advices

  • Using Stackdriver Logging and Monitoring
  • Use a lighter base image such as Alpine Linux improve deployment times

TerramEarth

Background

  • 20 million vehicles that collect 120 fields for data per second
  • data is stored locally
  • data is downloaded via a maintenanace port.
  • 200,000 vehicles are connected to a cellular network, allowing to collect data directly.

Existing technical environment

  • Linux and Windows-based systems that reside in a single U.S. west-coast-based data center. - - gzip CSV files from the field and upload via FTP
  • Place the data in their data warehouse.
  • Aggregated reports are based on data that is three weeks old.

Business requirements

  • Decrease unplanned vehicle downtime to less than one week
    • Use Streaming Pipeline
      • Cellular -> Cloud Pub/Sub -> Cloud Storage -> Cloud Dataflow -> BigQuery
    • Use Batch Pipeline
      • User -> Cloud Storage -> Cloud Dataflow -> BigQuery
  • Support the dealer network with more data on how their customers use their equipment to better position new products and services
    • Use Google Data Studio to create live charts that read directly from BigQuery. Give dealer representatives view rights to these charts to gain better understanding.
  • Have the ability to partner with different companies-especially with seed and fertilizer suppliers in the fast-growing agricultural business-to create compelling joint offerings for their customers

Technical requirements

  • Decrease latency to the American Midwest and East Coast
    • Upload data to regional bucket of location
  • Create a backup strategy
  • Increase security of data transfer from equipment to the data center
  • Improve data in the data warehouse
    • Use a BigQuery with table partitioning
  • Use customer and equipment data to anticipate customer needs

Applications

  • a custom python application reads uploaded files, then writes to the data warehouse.
  • a daily report to see what equipment needs repair.

Advices

  • Use Cloud Pub/Sub handles streaming data in real time
  • Use Cloud Data Studio to replace current analytics software
  • Use Cloud Endpoints manage and project their APIs

Dress4Win

Background

  • web app and mobile application
  • committing to a full migration to a public cloud
    • MySQL: Set up a MySQL replica/slave in Google Cloud using Cloud SQL and configure it for asynchronous replication from the MySQL master server on-premises until cutover.
    • Rabbit MQ: Cloud Pub/Sub
    • NAS: Cloud Storage bucket
    • SAN: Persistent disk
    • Apache Hadoop/Spark: Dataproc
  • moving their development and test environments
  • building a disaster recovery site

Technical environment

  • Databases:
    • MySQL. One server for user data, inventory, static data
      • Cloud SQL migration service
      • MySQL 5.7
      • 8 core CPUs
      • 128 GB of RAM
      • 2x 5TB HDD (RAID 1)
  • Compute
    • 40 web application servers providing micro-services based APIs and static content
      • Managed Instance group
      • Tomcat - Java
      • Nginx
      • Four core CPUs
      • 32 GB of RAM
    • 20 Apache Hadoop/Spark servers:
      • Migrate jobs to Dataproc
      • Data analysis
      • Real-time trending calculations
      • Eight core CPUs
      • 128 GB of RAM
      • 4x 5 TB HDD (RAID 1)
    • Three RabbitMQ servers for messaging, social notifications, and events:
      • Migrate to Cloud Pub/Sub
      • Eight core CPUs
      • 32GB of RAM
    • Miscellaneous servers:
      • Compute Engine
      • Jenkins, monitoring, bastion hosts, security scanners
      • Eight core CPUs
      • 32GB of RAM
  • Storage appliances:
    • iSCSI for VM hosts
    • Fibre channel SAN - MySQL databases
    • 1 PB total storage; 400 TB available
    • NAS - image storage, logs, backups
    • 100 TB total storage; 35 TB available

Business requirements

  • Build a reliable and reproducible environment with scaled parity of production
  • identity and access management (IAM) best practices for cloud to improve security
  • Improve business agility and speed of innovation through rapid provisioning of new resources Analyze and optimize architecture for performance

Technical requirements

  • Easily create non-production environments
    • Deployment Manager
  • Implement an automation framework for provisioning resources
    • Deployment Manager
  • Implement a continuous deployment process for deploying applications
    • pre-built Jenkins image form MarketPlace
  • Support failover of the production environment to cloud during an emergency
  • Encrypt data on the wire and at rest
  • Support multiple private connections between the production data center and cloud environment.
    • Dedicated Interconnect or Partner Interconnect