Microsoft Certified: Azure Data Engineer Associate

A candidate for the Azure Data Engineer Associate certification should have subject matter expertise integrating, transforming, and consolidating data from various structured and unstructured data systems into structures that are suitable for building analytics solutions.

Responsibilities for this role include helping stakeholders understand the data through exploration, building and maintaining secure and compliant data processing pipelines by using different tools and techniques. This professional uses various Azure data services and languages to store and produce cleansed and enhanced datasets for analysis.

Skills measured

Design and implement data storage
Design and develop data processing
Design and implement data security
Monitor and optimize data storage and data processing

Course Outline

Design a data storage structure

design an Azure Data Lake solution
recommend file types for storage
recommend file types for analytical queries
design for efficient querying
design for data pruning
design a folder structure that represents the levels of data transformation
design a distribution strategy
design a data archiving solution

Design a partition strategy

design a partition strategy for files
design a partition strategy for analytical workloads
design a partition strategy for efficiency/performance
design a partition strategy for Azure Synapse Analytics
identify when partitioning is needed in Azure Data Lake Storage Gen2

Design the serving layer

design star schemas
design slowly changing dimensions
design a dimensional hierarchy
design a solution for temporal data
design for incremental loading
design analytical stores
design metastores in Azure Synapse Analytics and Azure Databricks

Ingest and transform data

transform data by using Apache Spark
transform data by using Transact-SQL
transform data by using Data Factory
transform data by using Azure Synapse Pipelines
transform data by using Stream Analytics
cleanse data
split data
shred JSON
ncode and decode data
configure error handling for the transformation
normalize and denormalize values
transform data by using Scala
perform data exploratory analysis

Design and develop a stream processing solution

develop a stream processing solution by using Stream Analytics, Azure Databricks, and Azure Event Hubs
process data by using Spark structured streaming
monitor for performance and functional regressions
design and create windowed aggregates
handle schema drift
process time series data
process across partitions
process within one partition
configure checkpoints/watermarking during processing
scale resources
design and create tests for data pipelines
optimize pipelines for analytical or transactional purposes
handle interruptions
design and configure exception handling
upsert data
replay archived stream data
design a stream processing solution

Implement data security

implement data masking
encrypt data at rest and in motion
implement row-level and column-level security
implement Azure RBAC
implement POSIX-like ACLs for Data Lake Storage Gen
implement a data retention policy
implement a data auditing strategy
manage identities, keys, and secrets across different data platform technologies
implement secure endpoints (private and public)
implement resource tokens in Azure Databricks
load a DataFrame with sensitive informatio
write encrypted data to tables or Parquet files
manage sensitive information

Optimize and troubleshoot data storage and data processing

compact small files
rewrite user-defined functions (UDFs)
handle skew in data
handle data spill
tune shuffle partitions
find shuffling in a pipeline
optimize resource management
tune queries by using indexers
tune queries by using cache
optimize pipelines for analytical or transactional purposes
optimize pipeline for descriptive versus analytical workloads
troubleshoot a failed spark job
troubleshoot a failed pipeline run

About The Exam

This exam measures your ability to accomplish the following technical tasks: design and implement data storage; design and develop data processing; design and implement data security; and monitor and optimize data storage and data processing.