{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "07ebb5d3", "metadata": {}, "outputs": [], "source": [ "%%capture\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": null, "id": "b92ad576", "metadata": {}, "outputs": [], "source": [ "#| default_exp: export" ] }, { "cell_type": "code", "execution_count": null, "id": "96525ac4", "metadata": {}, "outputs": [], "source": [ "#| export\n", "import enum,pathlib,re\n", "from pathlib import Path\n", "from typing import IO\n", "\n", "import nbformat\n", "from nbformat import NotebookNode\n", "\n", "from nb.core import read_nb,CellType,filter_cells,mk_directive_re,strip_directives,PathOrFileLike" ] }, { "cell_type": "code", "execution_count": null, "id": "eb27bbd6", "metadata": {}, "outputs": [], "source": [ "import os,json,tempfile\n", "from io import StringIO\n", "from nb import test" ] }, { "cell_type": "markdown", "id": "a682b39f", "metadata": {}, "source": [ "# Exporting Notebooks" ] }, { "cell_type": "markdown", "id": "a21a7e2c", "metadata": {}, "source": [ "To support notebook-driven development, we need to be able to convert `ipynb` notebooks to `py` files. Without that, we wouldn't be able to create Python libraries that could be used by others or import code we want to use in one notebook from another notebook because it's not in a plain importable Python file.\n", "\n", "To export notebooks, we'll need to know:\n", "- What Python file are we exporting our notebook to?\n", "- Which cells are being exported to a Python file?\n", "\n", "The approach tools like [nbdev](https://nbdev.fast.ai/) or [nbexport](https://github.com/danhorner/nbexport) take is to have certain directives that determine the code export behavior. We'll use a similar approach, but we'll only support 2 directives for code export:\n", "- `#|default_exp: mymodule`: Set the output module for this notebook\n", "- `#|export`: Export this cell to the corresponding Python file" ] }, { "cell_type": "markdown", "id": "3b670537", "metadata": {}, "source": [ "The general approach we'll use for exporting notebooks to Python files is:\n", "1. Read in a `ipynb` notebook (`nb.core.read_nb`)\n", "2. Retrieve all code cells from the notebook (`nb.core.filter_cells`)\n", "3. Find all cells that should be exported:\n", " - Each notebook must have 1 (and only 1) `#| default_exp`. When a `#| default_exp` is found, it must be a single word (e.g. `mymodule`) or a dotted path (e.g. `mymodules.other_module`). If it is a dotted path, the submodules created should have `__init__.py` files created alongside them.\n", " - Any cells with an `#| export` directive should be marked for export to the generated Python file.\n", "4. Write out each cell sequentially into the Python file, stripping out any directives. When each cell is written out, it should have a comment above it describing the cell ID it came from." ] }, { "cell_type": "markdown", "id": "6f599791", "metadata": {}, "source": [ "## Directives" ] }, { "cell_type": "markdown", "id": "dfd6ac40", "metadata": {}, "source": [ "### Supporting the default_exp directive" ] }, { "cell_type": "code", "execution_count": null, "id": "ac846aaa", "metadata": {}, "outputs": [], "source": [ "#| export\n", "DIRECTIVE_DEFAULT_EXP_RE = mk_directive_re('default_exp', arg_re=r'(?P\\w+(\\.\\w+)*)')" ] }, { "cell_type": "code", "execution_count": null, "id": "5f890ed5", "metadata": {}, "outputs": [], "source": [ "matching_cells = [\n", " ('#| default_exp: bootstrap', 'bootstrap'), # Module name\n", " ('#| default_exp: bootstrap.another_module', 'bootstrap.another_module'), # Dotted module path\n", "]\n", "for line, module in matching_cells:\n", " match = DIRECTIVE_DEFAULT_EXP_RE.match(line)\n", " test.is_true(match)\n", " test.equal(match.group('module'), module)" ] }, { "cell_type": "code", "execution_count": null, "id": "d1f22bb8", "metadata": {}, "outputs": [], "source": [ "not_matching_cells = [\n", " '#| default_exp: ', # Missing module name\n", " '#| default_exp: !$', # Invalid module name\n", " '#| default_export: bootstrap', # Misspelled directive\n", " '#| default_exp: bootstrap.', # Dot without a following word\n", " '#| default_exp: bootstrap..', # Too many dots without a word in between\n", "]\n", "for line in not_matching_cells:\n", " test.is_false(DIRECTIVE_DEFAULT_EXP_RE.match(line))" ] }, { "cell_type": "markdown", "id": "50495b37", "metadata": {}, "source": [ "### Supporting the export directive" ] }, { "cell_type": "code", "execution_count": null, "id": "5b4e8fe6", "metadata": {}, "outputs": [], "source": [ "#| export\n", "DIRECTIVE_EXPORT_RE = mk_directive_re('export')" ] }, { "cell_type": "markdown", "id": "a7a778d2", "metadata": {}, "source": [ "### All Directives" ] }, { "cell_type": "code", "execution_count": null, "id": "828159b2", "metadata": {}, "outputs": [], "source": [ "#| export\n", "DIRECTIVES = [DIRECTIVE_DEFAULT_EXP_RE, DIRECTIVE_EXPORT_RE]" ] }, { "cell_type": "markdown", "id": "21e3d799", "metadata": {}, "source": [ "## Identifying Cells to Export" ] }, { "cell_type": "markdown", "id": "295b4a99", "metadata": {}, "source": [ "So, we should now have all the pieces to find all cells we should be exporting and the module we should be exporting to.\n", "\n", "**Algorithm:**\n", "1. Always check if there is a `default_exp` directive. There must be one (and only one) `default_exp` directive.\n", "3. Check for an explicit export directive." ] }, { "cell_type": "code", "execution_count": null, "id": "d1f32654", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def is_cell_exported(cell:NotebookNode)->bool:\n", " \"Check whether the code cell should be exported.\"\n", " return any(DIRECTIVE_EXPORT_RE.match(l) for l in cell['source'].splitlines())" ] }, { "cell_type": "code", "execution_count": null, "id": "a61db327", "metadata": {}, "outputs": [], "source": [ "test.is_true(is_cell_exported(nbformat.v4.new_code_cell('#| export')))\n", "test.is_true(is_cell_exported(nbformat.v4.new_code_cell('import re\\n#| export')))" ] }, { "cell_type": "code", "execution_count": null, "id": "b06462cc", "metadata": {}, "outputs": [], "source": [ "test.is_false(is_cell_exported(nbformat.v4.new_code_cell(('import re'))))\n", "test.is_false(is_cell_exported(nbformat.v4.new_code_cell(('1+1\\nfrom re import Pattern'))))" ] }, { "cell_type": "code", "execution_count": null, "id": "ddc5c9e6", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def get_default_export(cells:list[NotebookNode])->str:\n", " \"Retrieve the `default_exp` directive argument from a list of cells.\"\n", " matches = [\n", " m.group('module')\n", " for c in cells\n", " for l in c['source'].splitlines()\n", " if c['cell_type'] == CellType.CODE and (m:=DIRECTIVE_DEFAULT_EXP_RE.match(l))\n", " ]\n", " if len(matches) > 1: raise ValueError('More than one default_exp directive')\n", " if len(matches) == 0: raise ValueError('Missing default_exp directive')\n", " return matches[0]" ] }, { "cell_type": "code", "execution_count": null, "id": "7310f069", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook(cells=[nbformat.v4.new_code_cell('#| default_exp: export')])\n", "test.equal(get_default_export(nb.cells), 'export')" ] }, { "cell_type": "code", "execution_count": null, "id": "55fd5319", "metadata": {}, "outputs": [], "source": [ "with test.raises(exc=ValueError, contains='Missing default_exp directive'):\n", " get_default_export([nbformat.v4.new_code_cell('a=5')])" ] }, { "cell_type": "code", "execution_count": null, "id": "3da9dde8", "metadata": {}, "outputs": [], "source": [ "with test.raises(exc=ValueError, contains='More than one default_exp directive'):\n", " get_default_export([\n", " nbformat.v4.new_code_cell('#| default_exp: hello'),\n", " nbformat.v4.new_code_cell('#| default_exp: test'),\n", " ])" ] }, { "cell_type": "code", "execution_count": null, "id": "9ce855f0", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def find_cells_to_export(\n", " fp:PathOrFileLike|NotebookNode, # Where to read the notebook from.\n", ")->tuple[str,list[NotebookNode]]: # Returns the module name and list of cells to export.\n", " \"Find the code cells in an ipynb file to export out.\"\n", " if not isinstance(fp, NotebookNode): fp = read_nb(fp)\n", " code_cells = filter_cells(fp, CellType.CODE)\n", " module = get_default_export(code_cells)\n", " cells = [\n", " {**c, 'source': strip_directives(c['source'])}\n", " for c in code_cells\n", " if is_cell_exported(c)\n", " ]\n", " return module, cells" ] }, { "cell_type": "markdown", "id": "aa285ef2", "metadata": {}, "source": [ "If a notebook has a `default_exp` directive and exported cells, we should see those extracted out and the directive comments stripped out." ] }, { "cell_type": "code", "execution_count": null, "id": "ffaa9942", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook()\n", "nb.cells = [\n", " nbformat.v4.new_code_cell('#| default_exp: mymodule'),\n", " nbformat.v4.new_code_cell('#| export\\ndef hello(): return \"hi\"'),\n", "]\n", "test.equal(find_cells_to_export(nb), ('mymodule', [{**nb.cells[1], 'source': 'def hello(): return \"hi\"'}]))" ] }, { "cell_type": "markdown", "id": "85a57955", "metadata": {}, "source": [ "Additionally, we can pass in a path to a notebook rather than an in-memory notebook. If no cell is exported, the cells to export will be empty." ] }, { "cell_type": "code", "execution_count": null, "id": "329abbe7", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook(cells=[nbformat.v4.new_code_cell('#| default_exp: mymodule')])\n", "with tempfile.TemporaryDirectory() as tmp:\n", " p = tmp/Path('test.ipynb')\n", " p.write_text(json.dumps(nb))\n", " test.equal(find_cells_to_export(p), ('mymodule', []))" ] }, { "cell_type": "markdown", "id": "9e1d96de", "metadata": {}, "source": [ "An empty notebook should raise a `ValueError` because it doesn't have a `default_exp` directive." ] }, { "cell_type": "code", "execution_count": null, "id": "2c349b45", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook()\n", "with test.raises(ValueError): find_cells_to_export(nb)" ] }, { "cell_type": "markdown", "id": "66574d29", "metadata": {}, "source": [ "## Exporting a Notebook" ] }, { "cell_type": "code", "execution_count": null, "id": "3070f710", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def fmt_export_cell(cell:dict)->dict:\n", " \"Format a code cell's source with metadata to prepare for export.\"\n", " cell['source'] = f'# cell: {cell[\"id\"]}\\n' + cell['source'] + '\\n'\n", " return cell" ] }, { "cell_type": "code", "execution_count": null, "id": "e24f633f", "metadata": {}, "outputs": [], "source": [ "c = nbformat.v4.new_code_cell('a=5', id='1')\n", "test.equal(fmt_export_cell(c)['source'], '# cell: 1\\na=5\\n')" ] }, { "cell_type": "code", "execution_count": null, "id": "633e256b", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def mk_py_mod_scaffold(\n", " mod:str, # The Python module name or dotted path e.g. mod.mymod.file\n", " src_dir:str|Path, # The Python package source code directory scaffold\n", "):\n", " \"Create all intermediary __init__.py files and directories, if any, and the final .py file.\"\n", " if not mod: raise ValueError('The module path cannot be empty')\n", " src_dir = Path(src_dir)\n", " mod_file = (src_dir/Path(*mod.split('.'))).with_suffix('.py')\n", " mod_file.parent.mkdir(parents=True, exist_ok=True)\n", " mod_file.touch()\n", " pkg_dirs = list(mod_file.parents)\n", " for p in pkg_dirs[:pkg_dirs.index(src_dir) + 1]:\n", " (p / '__init__.py').touch()" ] }, { "cell_type": "code", "execution_count": null, "id": "e19b3289", "metadata": {}, "outputs": [], "source": [ "with tempfile.TemporaryDirectory() as tmp, test.raises(ValueError, contains='The module path cannot be empty'):\n", " mk_py_mod_scaffold('', src_dir=tmp)" ] }, { "cell_type": "code", "execution_count": null, "id": "c3a33968", "metadata": {}, "outputs": [], "source": [ "with tempfile.TemporaryDirectory() as tmp:\n", " mk_py_mod_scaffold('file', src_dir=tmp)\n", " test.equal(os.listdir(tmp), ['__init__.py', 'file.py'])" ] }, { "cell_type": "code", "execution_count": null, "id": "f2e3fcb4", "metadata": {}, "outputs": [], "source": [ "with tempfile.TemporaryDirectory() as tmp:\n", " mk_py_mod_scaffold('mod.file', src_dir=tmp)\n", " test.equal(os.listdir(tmp), ['__init__.py', 'mod'])\n", " test.equal(os.listdir(Path(tmp)/'mod'), ['__init__.py', 'file.py'])" ] }, { "cell_type": "code", "execution_count": null, "id": "f3ab7819", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def export_nb(\n", " fp:PathOrFileLike|NotebookNode, # The notebook to export.\n", " src_dir:str|Path='.', # The source directory to export the Python module to.\n", "):\n", " \"Export the exportable code cells to a Python file.\"\n", " module, cells = find_cells_to_export(fp)\n", " mk_py_mod_scaffold(module, src_dir)\n", " exp_path = (Path(src_dir)/module).with_suffix('.py')\n", " src = '\\n'.join(\n", " ['# AUTOGENERATED! DO NOT EDIT!\\n'] +\n", " list(map(lambda c: fmt_export_cell(c)['source'], cells))\n", " )\n", " Path(exp_path).write_text(src)" ] }, { "cell_type": "code", "execution_count": null, "id": "55f57bee", "metadata": {}, "outputs": [], "source": [ "def mk_nb(cells)->StringIO:\n", " nb = nbformat.v4.new_notebook(cells=cells)\n", " buf = StringIO()\n", " nbformat.write(nb, buf)\n", " buf.seek(0)\n", " return buf" ] }, { "cell_type": "code", "execution_count": null, "id": "31bb0f0b", "metadata": {}, "outputs": [], "source": [ "cells = [\n", " nbformat.v4.new_code_cell('#| default_exp: testmod'),\n", " nbformat.v4.new_code_cell('#| export \\nimport os'),\n", " nbformat.v4.new_code_cell('#| export\\ndef foo(): pass'),\n", " nbformat.v4.new_code_cell('import pathlib'),\n", "]\n", "nb = mk_nb(cells)\n", "src = f\"\"\"# AUTOGENERATED! DO NOT EDIT!\n", "\n", "# cell: {cells[1]['id']}\n", "import os\n", "\n", "# cell: {cells[2]['id']}\n", "def foo(): pass\n", "\"\"\"\n", "with tempfile.TemporaryDirectory() as tmp:\n", " export_nb(nb, src_dir=tmp)\n", " result = Path(tmp, 'testmod.py').read_text()\n", " test.equal(result, src)" ] }, { "cell_type": "code", "execution_count": null, "id": "8e1d4096", "metadata": {}, "outputs": [], "source": [ "export_nb('export.ipynb', src_dir='../nb/')" ] } ], "metadata": {}, "nbformat": 4, "nbformat_minor": 5 }