• Design and implement data storage
• Design and develop data processing
• Design and implement data security
• Monitor and optimize data storage and data processing

Course Outline

Design a data storage structure

• design an Azure Data Lake solution
• recommend file types for storage
• recommend file types for analytical queries
• design for efficient querying
• design for data pruning
• design a folder structure that represents the levels of data transformation
• design a distribution strategy
• design a data archiving solution

Design a partition strategy

• design a partition strategy for files
• design a partition strategy for analytical workloads
• design a partition strategy for efficiency/performance
• design a partition strategy for Azure Synapse Analytics
• identify when partitioning is needed in Azure Data Lake Storage Gen2

Design the serving layer

• design star schemas
• design slowly changing dimensions
• design a dimensional hierarchy
• design a solution for temporal data
• design for incremental loading
• design analytical stores
• design metastores in Azure Synapse Analytics and Azure Databricks

Ingest and transform data

• transform data by using Apache Spark
• transform data by using Transact-SQL
• transform data by using Data Factory
• transform data by using Azure Synapse Pipelines
• transform data by using Stream Analytics
• cleanse data
• split data
• shred JSON
• ncode and decode data
• configure error handling for the transformation
• normalize and denormalize values
• transform data by using Scala
• perform data exploratory analysis

Design and develop a stream processing solution

• develop a stream processing solution by using Stream Analytics, Azure Databricks, and Azure Event Hubs
• process data by using Spark structured streaming
• monitor for performance and functional regressions
• design and create windowed aggregates
• handle schema drift
• process time series data
• process across partitions
• process within one partition
• configure checkpoints/watermarking during processing
• scale resources
• design and create tests for data pipelines
• optimize pipelines for analytical or transactional purposes
• handle interruptions
• design and configure exception handling
• upsert data
• replay archived stream data
• design a stream processing solution

Implement data security

• implement data masking
• encrypt data at rest and in motion
• implement row-level and column-level security
• implement Azure RBAC
• implement POSIX-like ACLs for Data Lake Storage Gen
• implement a data retention policy
• implement a data auditing strategy
• manage identities, keys, and secrets across different data platform technologies
• implement secure endpoints (private and public)
• implement resource tokens in Azure Databricks
• load a DataFrame with sensitive informatio
• write encrypted data to tables or Parquet files
• manage sensitive information

Optimize and troubleshoot data storage and data processing

• compact small files
• rewrite user-defined functions (UDFs)
• handle skew in data
• handle data spill
• tune shuffle partitions
• find shuffling in a pipeline
• optimize resource management
• tune queries by using indexers
• tune queries by using cache
• optimize pipelines for analytical or transactional purposes
• optimize pipeline for descriptive versus analytical workloads
• troubleshoot a failed spark job
• troubleshoot a failed pipeline run

About The Exam