Jupyter Notebooks AI Skill Guide

Overview & Engine Architecture

Jupyter runs code cells against a kernel (usually IPython) with persistent in-memory state. The .ipynb JSON stores inputs, outputs, and metadata. Agents keep notebooks linear and restartable, parameterize with papermill when automating, and extract durable logic into .py modules once paths stabilize.

Notebook UI / VS Code / JupyterLab
      -> kernel (Python/R/Julia)
          -> cells mutate namespace
          -> outputs embedded in .ipynb

When to use this skill

• Exploratory analysis and teaching demos
• Lightweight reports with plots/tables
• Parameterized batch runs via papermill/nbconvert

Operational directives

1. "Restart kernel & run all" must succeed before sharing or committing.
2. Put secrets in env vars / keyring - never in cells that get committed.
3. Clear oversized outputs before git commits (or use nbstripout).
4. Import project modules instead of duplicating production ETL in cells.
5. Pin kernel/env (requirements.txt / conda lock) beside the notebook.

Minimal reproducible header

# cell 0
from pathlib import Path
import pandas as pd

DATA = Path("data/orders.parquet")
assert DATA.exists(), f"missing {DATA}"

df = pd.read_parquet(DATA)
df.head()

Papermill parameter cell

# tags: parameters
run_date = "2026-08-26"
input_path = "data/orders.parquet"

papermill analysis.ipynb out/analysis_2026-08-26.ipynb -p run_date 2026-08-26
jupyter nbconvert --to html out/analysis_2026-08-26.ipynb

Common failures

Symptom	Cause	Fix
Works only top-down once	hidden state / out-of-order run	restart & run all
Huge git diffs	embedded outputs/images	strip outputs; LFS if needed
Wrong package	multiple kernels/envs	select correct kernel; document
Kernel dies	memory spike	sample data; move to DuckDB/Polars

Best practices

• One narrative per notebook; split mega-notebooks by stage.
• Freeze random seeds when illustrating ML (@scikit-learn, @pytorch).
• Prefer Parquet inputs over pasted CSVs in repo.
• Use %matplotlib inline / explicit plt.show consistently for readers.

Limitations

• Not a production scheduler - wrap with @airflow / @prefect for pipelines.
• Collaborative merge conflicts on .ipynb are painful; prefer scripts for shared logic.
• Remote kernels and HPC proxies need local IT/docs.

Related skills

• @pandas / @polars / @duckdb - compute inside cells
• @mlflow - log notebook experiments
• @great-expectations - validate data before analysis