ADE Python for Data Engineers

Python connects data engineers to every corner of Azure. Databricks notebooks, Azure Functions, and Fabric notebooks all run it. A small set of Python skills covers most pipeline tasks: reading files, cleaning values, and writing results back to storage.

Why Python Fits Data Work

Python reads like plain English, and thousands of free libraries exist for data tasks. The pandas library handles tables, PySpark handles huge data, and the Azure SDK talks to cloud services. One language covers small scripts and big pipelines.

Core Building Blocks

Variables and Types

A variable is a labeled box that holds a value. Python guesses the type from the value.

city = "Pune"          # text
rainfall_mm = 12.5     # decimal number
station_count = 4      # whole number
is_active = True       # true or false

Lists and Dictionaries

A list holds an ordered group of items. A dictionary holds named values, much like a contact card with labeled fields.

cities = ["Pune", "Delhi", "Goa"]
station = {"id": "S101", "city": "Pune", "active": True}
print(station["city"])

Loops and Functions

A loop repeats work for each item. A function packages a task under a name so any pipeline can reuse it.

def to_fahrenheit(celsius):
    return celsius * 9 / 5 + 32

for temp in [20, 25, 30]:
    print(to_fahrenheit(temp))

Scenario: Weather Station Cleanup

A fictional network of weather stations drops a CSV file into Azure Data Lake Storage Gen2 every night. The file holds temperature readings with missing values and duplicate rows. A Python script cleans the file and saves a compact Parquet copy.

Raw CSV in lake ---> Read with pandas ---> Drop blanks ---> Remove duplicates
                                                                |
                                                                v
            Report-ready Parquet in lake <--- Average per station

import pandas as pd

df = pd.read_csv("readings.csv")
df = df.dropna(subset=["temperature"])
df = df.drop_duplicates(subset=["station_id", "reading_time"])

daily = (
    df.groupby("station_id", as_index=False)["temperature"]
      .mean()
      .rename(columns={"temperature": "avg_temperature"})
)

daily.to_parquet("daily_avg.parquet", index=False)

The script reads the CSV, removes unusable rows, averages temperature per station, and writes Parquet. The pyarrow library supplies the Parquet support behind the scenes.

Handling Errors Gracefully

Files go missing and networks fail. A try block catches problems so the pipeline can log a clear message instead of crashing in silence.

try:
    df = pd.read_csv("readings.csv")
except FileNotFoundError:
    print("Input file missing. Pipeline stops with a clear message.")
    raise

Talking to Azure from Python

Two packages cover most storage work. The azure-identity package signs in with a managed identity, and azure-storage-file-datalake reads and writes files in the lake.

from azure.identity import DefaultAzureCredential
from azure.storage.filedatalake import DataLakeServiceClient

credential = DefaultAzureCredential()
service = DataLakeServiceClient(
    account_url="https://mylake.dfs.core.windows.net",
    credential=credential
)
fs = service.get_file_system_client("raw")
for path in fs.get_paths("weather"):
    print(path.name)

DefaultAzureCredential picks up the identity of the running service, so the code carries no passwords.

Reading JSON from an API

Many sources return JSON. The requests library fetches the response, and pandas flattens the nested fields into a table.

import requests
import pandas as pd

response = requests.get("https://example.com/api/stations", timeout=30)
response.raise_for_status()
stations = pd.json_normalize(response.json())
print(stations.head())

The timeout setting stops a slow server from freezing the pipeline. The raise_for_status call turns a bad response into a visible error right away.

Working with Dates

Dates arrive as text in countless shapes. Convert them to a real date type early, and every later step becomes simpler.

df["reading_time"] = pd.to_datetime(df["reading_time"], errors="coerce")
df["reading_date"] = df["reading_time"].dt.date

The errors="coerce" setting turns unreadable dates into blanks that a cleanup step can catch.

Pandas or PySpark

Pandas loads all data into the memory of one machine. A file of a few million rows fits comfortably. A table of billions of rows needs PySpark, which splits the work across many machines. The code style looks similar in both, so the pandas habits learned here transfer directly.

Where Python Runs in Azure

Place	Typical Use
Databricks notebook	Large transformations with PySpark
Fabric notebook	Lakehouse exploration and cleaning
Azure Functions	Small event-driven tasks, such as reacting to a new file
Local machine	Development and quick tests

Good Habits

• Keep every function short and focused on one task.
• Store settings in parameters, never inside the code.
• Log row counts at every step to spot surprises.
• Use pandas for data that fits in memory and PySpark for anything larger.

Key Points

• Python plus pandas handles cleaning and reshaping of small to medium data.
• Functions and loops turn repeated work into reusable code.
• The Azure SDK with managed identity reads and writes lake files safely.
• Error handling keeps pipelines honest and easy to debug.