Fabric Notebook Performance Checklist

Use this checklist with a representative run and comparable input volume. Record session acquisition, major Spark stages, and output writing separately. A shorter total duration is useful only when row counts, business grain, and rerun behavior remain correct.

Before running

• Confirm the notebook has one clear transformation purpose and owner.
• Record the input boundary, expected output grain, and rerun behavior.
• Remove unused imports and avoid unnecessary inline package installation.
• Confirm whether starter, reused high-concurrency, or isolated compute fits the workload.
• Parameterize environment-specific Lakehouses, dates, paths, and correlation IDs.
• Capture a baseline using representative volume before changing configuration.

While reading

• Select only columns required by the transformation and its quality checks.
• Apply selective filters close to the source and confirm them in the executed plan.
• Avoid reading complete history for an incremental run.
• Check partition pruning, Delta data skipping, input file count, and small-file patterns.
• Measure rows or bytes entering the first wide transformation.

While transforming

• Avoid repeated actions such as multiple counts or displays on the same long lineage.
• Do not use collect() or toPandas() for results that may exceed driver memory.
• Identify shuffle-heavy joins, groups, sorts, distinct operations, and repartitioning.
• Check task duration and partition sizes for skew rather than tuning one global setting blindly.
• Broadcast only measured small reference data.
• Cache only expensive DataFrames reused by multiple actions; materialize and unpersist them deliberately.

While writing

• Choose append, overwrite, or merge from the data contract and rerun requirements.
• Avoid unnecessary repartitioning, especially forcing all output through one partition.
• Monitor file count and write-heavy stages; compact according to downstream read patterns.
• Evaluate Optimize Write, V-Order, Z-Order, and table maintenance by workload and consumer.
• Validate output row counts, keys, partitions, and important measures.
• Confirm a retry cannot duplicate or partially replace the target unexpectedly.

Pipeline and orchestration

• Enable and use session tags for compatible pipeline notebooks when reuse is appropriate.
• Keep stable session tags separate from per-run correlation identifiers.
• Separate incompatible Lakehouses, environments, compute configurations, identities, or resource-heavy workloads.
• Log pipeline activity, notebook, acquisition, first action, stage, and write durations.
• Use Fabric monitoring and session details to identify the true slow layer.
• Change one understood variable, rerun comparable data, and retain correctness controls.