{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "eae7a190", "metadata": {}, "outputs": [], "source": [ "%%capture\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": null, "id": "ce317915", "metadata": {}, "outputs": [], "source": [ "#| default_exp: clean" ] }, { "cell_type": "code", "execution_count": null, "id": "189df365", "metadata": {}, "outputs": [], "source": [ "#| export\n", "from nbformat import NotebookNode\n", "\n", "from nb.core import read_nb,PathOrFileLike" ] }, { "cell_type": "code", "execution_count": null, "id": "6a4b558e", "metadata": {}, "outputs": [], "source": [ "import copy\n", "import nbformat\n", "from nb import test" ] }, { "cell_type": "markdown", "id": "c3022df1", "metadata": {}, "source": [ "# Clean\n", "\n", "> Cleaning ipynb notebooks to minimize diffs across machines" ] }, { "cell_type": "markdown", "id": "ecefd668", "metadata": {}, "source": [ "One of the inherent challenges with Jupyter notebooks is they can have noisy diffs when you use them with version control systems. This can make it harder to tell what has actually changed when reviewing.\n", "\n", "To alleviate this problem, we can 'clean' the notebooks before we check them in." ] }, { "cell_type": "markdown", "id": "4af091bd", "metadata": {}, "source": [ "If a notebook cell is a code cell, then it will have an 'outputs' section. Some of these output cells have 'execution_count's, which we want to clear because it does not matter how many times a cell was executed when reviewing a change." ] }, { "cell_type": "code", "execution_count": null, "id": "b436a7c3", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def _clean_cell_output(cell:NotebookNode)->NotebookNode:\n", " for o in cell.get('outputs', []):\n", " if 'execution_count' in o:\n", " o['execution_count'] = None\n", " return cell" ] }, { "cell_type": "code", "execution_count": null, "id": "0823ae66", "metadata": {}, "outputs": [], "source": [ "o = nbformat.v4.new_output('execute_result', data={'text/plain': '42'}, execution_count=5)\n", "cc = nbformat.v4.new_code_cell(source=[], outputs=[o])\n", "cleaned = _clean_cell_output(copy.deepcopy(cc))\n", "test.equal(cleaned, {**cc, 'outputs': [{**o, 'execution_count': None}]})" ] }, { "cell_type": "code", "execution_count": null, "id": "4c127179", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def _clean_cell(cell:NotebookNode):\n", " if 'execution_count' in cell: cell['execution_count'] = None\n", " if cell['source'] == '': cell['source'] = []\n", " cell['metadata'] = {}\n", " return _clean_cell_output(cell)" ] }, { "cell_type": "code", "execution_count": null, "id": "1256ed4b", "metadata": {}, "outputs": [], "source": [ "cc = nbformat.v4.new_code_cell(\"print('hello')\", execution_count=10)\n", "cleaned = _clean_cell(copy.deepcopy(cc))\n", "test.equal(cleaned, {**cc, 'execution_count': None})" ] }, { "cell_type": "code", "execution_count": null, "id": "e772071d", "metadata": {}, "outputs": [], "source": [ "cc = nbformat.v4.new_code_cell(\"\")\n", "cleaned = _clean_cell(copy.deepcopy(cc))\n", "test.equal(cleaned, {**cc, 'source': []})" ] }, { "cell_type": "code", "execution_count": null, "id": "ef055ecb", "metadata": {}, "outputs": [], "source": [ "cc = nbformat.v4.new_code_cell(\"print('hello')\", metadata={\"some_key\": 1})\n", "cleaned = _clean_cell(copy.deepcopy(cc))\n", "test.equal(cleaned, {**cc, 'metadata': {}})" ] }, { "cell_type": "code", "execution_count": null, "id": "944a371b", "metadata": {}, "outputs": [], "source": [ "#| export\n", "_allowed_nb_metadata_keys = {\"kernelspec\", \"jupytext\", \"widgets\"}" ] }, { "cell_type": "code", "execution_count": null, "id": "00c3e419", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def clean_nb(nb:PathOrFileLike|NotebookNode)->NotebookNode:\n", " \"Clean noisy metadata in the notebook.\"\n", " if not isinstance(nb, NotebookNode): nb = read_nb(nb)\n", " for c in nb['cells']: _clean_cell(c)\n", " nb['metadata'] = {k:v for k,v in nb['metadata'].items() if k in _allowed_nb_metadata_keys}\n", " if nb['metadata'].get('kernelspec', {}).get('name'):\n", " nb['metadata']['kernelspec']['display_name'] = nb[\"metadata\"][\"kernelspec\"][\"name\"]\n", " return nb" ] }, { "cell_type": "code", "execution_count": null, "id": "df8e33bf", "metadata": {}, "outputs": [], "source": [ "import nb; nb.export_nb('clean.ipynb', src_dir='../nb/')" ] } ], "metadata": { "kernelspec": { "display_name": "python3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 5 }