{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "b92ad576", "metadata": {}, "outputs": [], "source": [ "#| default_exp: export" ] }, { "cell_type": "code", "execution_count": null, "id": "96525ac4", "metadata": {}, "outputs": [], "source": [ "#| export\n", "import enum,pathlib,re\n", "from pathlib import Path\n", "from typing import IO\n", "\n", "import nbformat\n", "from nbformat import NotebookNode" ] }, { "cell_type": "code", "execution_count": null, "id": "eb27bbd6", "metadata": {}, "outputs": [], "source": [ "import os,json,tempfile\n", "from io import StringIO" ] }, { "cell_type": "markdown", "id": "a682b39f", "metadata": {}, "source": [ "# Exporting Notebooks to Python Files" ] }, { "cell_type": "markdown", "id": "a21a7e2c", "metadata": {}, "source": [ "To get our notebook-driven source code bootstrapped, we'll first need to be able to convert `ipynb` files to `py` files. Without that, we'd be unable to later import code we want to use in one notebook from another notebook because it's not in a plain importable Python file yet.\n", "\n", "To accomplish that, for any given code well, we'll need to know:\n", "- What Python file are we exporting our notebook to?\n", "- Which cells are we exporting to our Python file?\n", "\n", "The approach tools like [nbdev](https://nbdev.fast.ai/) or [nbexport](https://github.com/danhorner/nbexport) take is to have certain directives that determine the code export behavior:\n", "- `#|default_exp: mymodule`: Set the output module for this notebook\n", "- `#|export`: Export this cell to the corresponding Python file" ] }, { "cell_type": "markdown", "id": "3b670537", "metadata": {}, "source": [ "To export a notebook, we'll need to:\n", "1. Read in a ipynb notebook\n", "2. Retrieve all code cells from the notebook\n", "3. Find all cells that should be exported:\n", " - Each notebook must have 1 (and only 1) `#| default_exp`. When a `#| default_exp` is found, it must be a single word e.g. `mymodule` or a dotted path e.g. `mymodules.other_module`. If it is a dotted path, the submodules created should have `__init__.py` files also created alongside them.\n", " - Any cells with an `#| export` explicit directive should be marked for export.\n", "4. Write out each cell sequentially into the Python file, stripping out any directives. When each cell is written out, it should have a comment above it describing the cell ID it came from." ] }, { "cell_type": "markdown", "id": "f1c95690", "metadata": {}, "source": [ "## Reading Notebooks" ] }, { "cell_type": "markdown", "id": "037fb28c", "metadata": {}, "source": [ "To read in `ipynb` files, we can use a library provided Jupyter called `nbformat`. And simply because we'll be reading in this notebook as our working copy while developing this export behavior, let's add a raw cell just so we have an example of all 3 cell types." ] }, { "cell_type": "raw", "id": "d9f3cb10", "metadata": {}, "source": [ "This is a raw cell just to have an example." ] }, { "cell_type": "markdown", "id": "4c0671ed", "metadata": {}, "source": [ "Now, we can read in our current notebook:" ] }, { "cell_type": "code", "execution_count": null, "id": "29a65b22", "metadata": {}, "outputs": [], "source": [ "#| export\n", "type PathOrFileLike = str|Path|IO[str]" ] }, { "cell_type": "code", "execution_count": null, "id": "27202360", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def read_nb(\n", " fp:PathOrFileLike, # Where to read the notebook from.\n", " v:int=4, # The notebook format major version to read the notebook as.\n", ")->NotebookNode:\n", " \"Read in a notebook as a dictionary-like NotebookNode.\"\n", " return nbformat.read(fp, as_version=v)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a689a07b", "metadata": {}, "outputs": [], "source": [ "for fp in ['export.ipynb', Path('export.ipynb')]:\n", " nb = read_nb(fp)\n", " assert list(nb.keys()) == ['cells', 'metadata', 'nbformat', 'nbformat_minor']\n", " assert type(nb) == NotebookNode" ] }, { "cell_type": "markdown", "id": "7d133fff", "metadata": {}, "source": [ "We can see that the top-level format of a notebook is:\n", "- `cells`: A list of objects for each cell in the notebook.\n", "- `metadata`: Free-form notebook level metadata used by tools and editors.\n", "- `nbformat`: Notebook format major version\n", "- `nbformat_minor`: Notebook format minor version" ] }, { "cell_type": "markdown", "id": "10a6afcb", "metadata": {}, "source": [ "Within cells, there are generally 3 types of cells:\n", "- `code`\n", "- `markdown`\n", "- `raw`" ] }, { "cell_type": "code", "execution_count": null, "id": "1ee0d459", "metadata": {}, "outputs": [], "source": [ "#| export\n", "class CellType(enum.StrEnum):\n", " \"Possible Jupyter notebook cell types.\"\n", " CODE = 'code'\n", " MD = 'markdown'\n", " RAW = 'raw'" ] }, { "cell_type": "markdown", "id": "46ed0445", "metadata": {}, "source": [ "## Retrieving All Code Cells" ] }, { "cell_type": "code", "execution_count": null, "id": "133ebd88", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def filter_cells(\n", " nb:NotebookNode, # A notebook that has been read in via `read_nb`.\n", " ct:CellType, # The cells to filter down to.\n", ")->list[NotebookNode]:\n", " \"Retrieve specific cell types from a notebook.\"\n", " return list(filter(lambda c: c['cell_type'] == ct, nb['cells']))" ] }, { "cell_type": "code", "execution_count": null, "id": "76917bc5", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'cell_type': 'code',\n", " 'execution_count': None,\n", " 'id': 'b92ad576',\n", " 'metadata': {},\n", " 'outputs': [],\n", " 'source': '#| default_exp: export'}" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "code_cell = filter_cells(nb, CellType.CODE)[0]\n", "assert list(code_cell.keys()) == ['cell_type', 'execution_count', 'id', 'metadata', 'outputs', 'source']\n", "assert code_cell['cell_type'] == CellType.CODE\n", "code_cell" ] }, { "cell_type": "code", "execution_count": null, "id": "4186fffe", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'cell_type': 'markdown',\n", " 'id': 'a682b39f',\n", " 'metadata': {},\n", " 'source': '# Exporting Notebooks to Python Files'}" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "md_cell = filter_cells(nb, CellType.MD)[0]\n", "assert list(md_cell.keys()) == ['cell_type', 'id', 'metadata', 'source']\n", "assert md_cell['cell_type'] == CellType.MD\n", "md_cell" ] }, { "cell_type": "code", "execution_count": null, "id": "46e56abf", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'cell_type': 'raw',\n", " 'id': 'd9f3cb10',\n", " 'metadata': {},\n", " 'source': 'This is a raw cell just to have an example.'}" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "raw_cell = filter_cells(nb, CellType.RAW)[0]\n", "assert list(raw_cell.keys()) == ['cell_type', 'id', 'metadata', 'source']\n", "assert raw_cell['cell_type'] == CellType.RAW\n", "raw_cell" ] }, { "cell_type": "markdown", "id": "4852b261", "metadata": {}, "source": [ "Within each cell object, they will all have:\n", "- `id`: A unique identifier for that cell.\n", "- `cell_type`: One of the 3 cell type names.\n", "- `metadata`: Free-form metadata for that cell used by various tools/editors.\n", "- `source`: The content of the cell, whether it is prose or code.\n", "\n", "Additionally, `code` cell types will have:\n", "- `outputs`: The stdout/stderr of the cell after the code is executed.\n", "- `execution_count`: The number of times the cell was run." ] }, { "cell_type": "markdown", "id": "6b7469e3", "metadata": {}, "source": [ "## Find Code Cells to Export" ] }, { "cell_type": "markdown", "id": "798249d5", "metadata": {}, "source": [ "### Matching Directives" ] }, { "cell_type": "markdown", "id": "f671cafa", "metadata": {}, "source": [ "A regular expression ran against each line in a cell should allow us to find each directive we support. Since we'll need to re-create this regression expression for a few directives, we'll create a small helper." ] }, { "cell_type": "code", "execution_count": null, "id": "83a898c2", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def _mk_directive_re(\n", " name:str, # The name of the directive\n", " arg_re:str|None=None, # A regex pattern to match an argument for the directive\n", ")->re.Pattern:\n", " \"Compile a regex to match a directive, optionally supporting an argument.\"\n", " arg = rf':\\s*{arg_re}' if arg_re else ''\n", " return re.compile(rf'^\\s*#\\|\\s*{name}{arg}\\s*$')" ] }, { "cell_type": "code", "execution_count": null, "id": "698d769b", "metadata": {}, "outputs": [], "source": [ "_TEST_DIRECTIVE_RE = _mk_directive_re('test')" ] }, { "cell_type": "code", "execution_count": null, "id": "f6d654fd", "metadata": {}, "outputs": [], "source": [ "test_dir_match = [\n", " ' #| test \\t \\n', # Lots of whitespace around each token\n", " '#|test', # No whitespace around each token\n", "]\n", "for line in test_dir_match:\n", " assert _TEST_DIRECTIVE_RE.match(line), f\"No match: {line!r}\"" ] }, { "cell_type": "code", "execution_count": null, "id": "02f06ee6", "metadata": {}, "outputs": [], "source": [ "test_dir_no_match = [\n", " '| test', # Missing #\n", " '# test', # Missing |\n", " '#| testt', # Misspelled directive\n", " '#| test test', # Extra word after directive\n", "]\n", "for line in test_dir_no_match:\n", " assert not _TEST_DIRECTIVE_RE.match(line), f\"Unexpected match: {line!r}\"" ] }, { "cell_type": "code", "execution_count": null, "id": "285e2320", "metadata": {}, "outputs": [], "source": [ "_TEST_ARG_DIRECTIVE_RE = _mk_directive_re('test', arg_re=r'\\w')" ] }, { "cell_type": "code", "execution_count": null, "id": "83fd068c", "metadata": {}, "outputs": [], "source": [ "test_dir_arg_match = [\n", " ' #| test: \\t a \\n', # Lots of whitespace around each token\n", " '#|test:b', # No whitespace around each token\n", "]\n", "for line in test_dir_arg_match:\n", " assert _TEST_ARG_DIRECTIVE_RE.match(line), f\"No match: {line!r}\"" ] }, { "cell_type": "code", "execution_count": null, "id": "dd9945ab", "metadata": {}, "outputs": [], "source": [ "test_arg_dir_no_match = [\n", " '#| test a', # Missing :\n", " '#| test: ', # Missing arg\n", " '#| test: aa', # Invalid arg (should be only 1 letter)\n", "]\n", "for line in test_dir_no_match + test_arg_dir_no_match:\n", " assert not _TEST_ARG_DIRECTIVE_RE.match(line), f\"Unexpected match: {line!r}\"" ] }, { "cell_type": "markdown", "id": "dfd6ac40", "metadata": {}, "source": [ "#### Supporting the default_exp directive" ] }, { "cell_type": "code", "execution_count": null, "id": "ac846aaa", "metadata": {}, "outputs": [], "source": [ "#| export\n", "DIRECTIVE_DEFAULT_EXP_RE = _mk_directive_re('default_exp', arg_re=r'(?P\\w+(\\.\\w+)*)')" ] }, { "cell_type": "code", "execution_count": null, "id": "5f890ed5", "metadata": {}, "outputs": [], "source": [ "matching_cells = [\n", " ('#| default_exp: bootstrap', 'bootstrap'), # Module name\n", " ('#| default_exp: bootstrap.another_module', 'bootstrap.another_module'), # Dotted module path\n", "]\n", "for line, module in matching_cells:\n", " match = DIRECTIVE_DEFAULT_EXP_RE.match(line)\n", " assert match, f\"No match: {line!r}\"\n", " assert match.group('module') == module" ] }, { "cell_type": "code", "execution_count": null, "id": "d1f22bb8", "metadata": {}, "outputs": [], "source": [ "not_matching_cells = [\n", " '#| default_exp: ', # Missing module name\n", " '#| default_exp: !$', # Invalid module name\n", " '#| default_export: bootstrap', # Misspelled directive\n", " '#| default_exp: bootstrap.', # Dot without a following word\n", " '#| default_exp: bootstrap..', # Too many dots without a word in between\n", "]\n", "for line in not_matching_cells:\n", " assert not DIRECTIVE_DEFAULT_EXP_RE.match(line), f\"Unexpected match: {line!r}\"" ] }, { "cell_type": "markdown", "id": "50495b37", "metadata": {}, "source": [ "#### Supporting the export directive" ] }, { "cell_type": "code", "execution_count": null, "id": "5b4e8fe6", "metadata": {}, "outputs": [], "source": [ "#| export\n", "DIRECTIVE_EXPORT_RE = _mk_directive_re('export')" ] }, { "cell_type": "markdown", "id": "a7a778d2", "metadata": {}, "source": [ "#### All Directives" ] }, { "cell_type": "code", "execution_count": null, "id": "828159b2", "metadata": {}, "outputs": [], "source": [ "#| export\n", "DIRECTIVES = [DIRECTIVE_DEFAULT_EXP_RE, DIRECTIVE_EXPORT_RE]" ] }, { "cell_type": "markdown", "id": "21e3d799", "metadata": {}, "source": [ "### Identifying Cells to Export" ] }, { "cell_type": "markdown", "id": "295b4a99", "metadata": {}, "source": [ "So, we should now have all the pieces to determine find all cells we should be exporting and the module we should be exporting to.\n", "\n", "**Algorithm:**\n", "1. Always check if there is a `default_exp` directive. There must be one (and only one) `default_exp` directive.\n", "3. Check for an explicit export directive." ] }, { "cell_type": "code", "execution_count": null, "id": "d327883e", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def has_directive(src:str)->bool:\n", " \"Check whether a line or block of source code has a nb directive.\"\n", " return any(any(r.match(l) for r in DIRECTIVES) for l in src.splitlines())" ] }, { "cell_type": "code", "execution_count": null, "id": "6ea5e776", "metadata": {}, "outputs": [], "source": [ "assert has_directive('#| default_exp: bootstrap')\n", "assert has_directive('#| export')\n", "assert has_directive('max(1,20)\\n#| export')\n", "assert not has_directive('# comment\\nmax(1, 20)\\n')" ] }, { "cell_type": "code", "execution_count": null, "id": "b6f47497", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def strip_directives(src:str|list[str])->str:\n", " \"Remove all nb directives from the source code.\"\n", " src = src if isinstance(src, list) else src.splitlines()\n", " return \"\\n\".join(filter(lambda l: not has_directive(l), src))" ] }, { "cell_type": "code", "execution_count": null, "id": "7acbb8ff", "metadata": {}, "outputs": [], "source": [ "assert strip_directives('#| export\\na=5') == 'a=5'\n", "assert strip_directives('#| default_exp: bootstrap\\na=5') == 'a=5'\n", "assert strip_directives(['a=10', 'a+=10', '#|export']) == 'a=10\\na+=10'" ] }, { "cell_type": "code", "execution_count": null, "id": "d1f32654", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def is_cell_exported(cell:NotebookNode)->bool:\n", " \"Check whether the code cell should be exported.\"\n", " return any(DIRECTIVE_EXPORT_RE.match(l) for l in cell['source'].splitlines())" ] }, { "cell_type": "code", "execution_count": null, "id": "64175ad3", "metadata": {}, "outputs": [], "source": [ "mk_cell = lambda s,id=1: dict(id=id,source=s,cell_type=CellType.CODE)\n", "assert is_cell_exported(mk_cell('#| export'))\n", "assert is_cell_exported(mk_cell('import re\\n#| export'))\n", "assert not is_cell_exported(mk_cell('import re'))\n", "assert not is_cell_exported(mk_cell('1+1\\nfrom re import Pattern'))" ] }, { "cell_type": "code", "execution_count": null, "id": "ddc5c9e6", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def get_default_export(cells:list[NotebookNode])->str:\n", " \"Retrieve the `default_exp` directive argument from a list of cells.\"\n", " matches = [\n", " m.group('module')\n", " for c in cells\n", " for l in c['source'].splitlines()\n", " if c['cell_type'] == CellType.CODE and (m:=DIRECTIVE_DEFAULT_EXP_RE.match(l))\n", " ]\n", " if len(matches) > 1: raise ValueError('More than one default_exp directive')\n", " if len(matches) == 0: raise ValueError('Missing default_exp directive')\n", " return matches[0]" ] }, { "cell_type": "code", "execution_count": null, "id": "7310f069", "metadata": {}, "outputs": [], "source": [ "assert get_default_export(read_nb('export.ipynb')['cells']) == 'export'\n", "\n", "try: get_default_export([mk_cell('a=5')]); assert False, \"Expected ValueError\"\n", "except ValueError: pass\n", "\n", "try:\n", " get_default_export([mk_cell('#| default_exp: hello'), mk_cell('#| default_exp: test')])\n", " assert False, \"Expected ValueError\"\n", "except ValueError: pass" ] }, { "cell_type": "code", "execution_count": null, "id": "9ce855f0", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def find_cells_to_export(\n", " fp:PathOrFileLike|NotebookNode, # Where to read the notebook from.\n", ")->tuple[str,list[NotebookNode]]: # Returns the module name and list of cells to export.\n", " \"Find the code cells in an ipynb file to export out.\"\n", " if not isinstance(fp, NotebookNode): fp = read_nb(fp)\n", " code_cells = filter_cells(fp, CellType.CODE)\n", " module = get_default_export(code_cells)\n", " cells = [\n", " {**c, 'source': strip_directives(c['source'])}\n", " for c in code_cells\n", " if is_cell_exported(c)\n", " ]\n", " return module, cells" ] }, { "cell_type": "markdown", "id": "aa285ef2", "metadata": {}, "source": [ "If a notebook has a `default_exp` directive and exported cells, we should see those extracted out and the directive comments stripped out." ] }, { "cell_type": "code", "execution_count": null, "id": "ffaa9942", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook()\n", "nb.cells = [\n", " nbformat.v4.new_code_cell('#| default_exp: mymodule'),\n", " nbformat.v4.new_code_cell('#| export\\ndef hello(): return \"hi\"'),\n", "]\n", "assert find_cells_to_export(nb) == ('mymodule', [{**nb.cells[1], 'source': 'def hello(): return \"hi\"'}])" ] }, { "cell_type": "markdown", "id": "85a57955", "metadata": {}, "source": [ "Additionally, we can pass in a path to a notebook rather than an in-memory notebook. If no cell is exported, the cells to export will be empty." ] }, { "cell_type": "code", "execution_count": null, "id": "329abbe7", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook(cells=[nbformat.v4.new_code_cell('#| default_exp: mymodule')])\n", "with tempfile.TemporaryDirectory() as tmp:\n", " p = tmp/Path('test.ipynb')\n", " p.write_text(json.dumps(nb))\n", " assert find_cells_to_export(p) == ('mymodule', [])" ] }, { "cell_type": "markdown", "id": "9e1d96de", "metadata": {}, "source": [ "An empty notebook should raise a `ValueError` because it doesn't have a `default_exp` directive." ] }, { "cell_type": "code", "execution_count": null, "id": "2c349b45", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook()\n", "try: find_cells_to_export(nb); assert False\n", "except ValueError: pass" ] }, { "cell_type": "markdown", "id": "66574d29", "metadata": {}, "source": [ "## Exporting a Notebook" ] }, { "cell_type": "code", "execution_count": null, "id": "3070f710", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def fmt_export_cell(cell:dict)->dict:\n", " \"Format a code cell's source with metadata to prepare for export.\"\n", " cell['source'] = f'# cell: {cell[\"id\"]}\\n' + cell['source'] + '\\n'\n", " return cell" ] }, { "cell_type": "code", "execution_count": null, "id": "e24f633f", "metadata": {}, "outputs": [], "source": [ "assert fmt_export_cell(mk_cell('a=5', id=1))['source'] == '# cell: 1\\na=5\\n'" ] }, { "cell_type": "code", "execution_count": null, "id": "633e256b", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def mk_py_mod_scaffold(\n", " mod:str, # The Python module name or dotted path e.g. mod.mymod.file\n", " src_dir:str|Path, # The Python package source code directory scaffold\n", "):\n", " \"Create all intermediary __init__.py files and directories, if any, and the final .py file.\"\n", " if not mod: raise ValueError('The module path cannot be empty')\n", " src_dir = Path(src_dir)\n", " mod_file = (src_dir/Path(*mod.split('.'))).with_suffix('.py')\n", " mod_file.parent.mkdir(parents=True, exist_ok=True)\n", " mod_file.touch()\n", " pkg_dirs = list(mod_file.parents)\n", " for p in pkg_dirs[:pkg_dirs.index(src_dir) + 1]:\n", " (p / '__init__.py').touch()" ] }, { "cell_type": "code", "execution_count": null, "id": "e19b3289", "metadata": {}, "outputs": [], "source": [ "with tempfile.TemporaryDirectory() as tmp:\n", " try: mk_py_mod_scaffold('', src_dir=tmp); assert False, \"Should raise ValueError\"\n", " except ValueError: pass\n", " assert os.listdir(tmp) == []" ] }, { "cell_type": "code", "execution_count": null, "id": "c3a33968", "metadata": {}, "outputs": [], "source": [ "with tempfile.TemporaryDirectory() as tmp:\n", " mk_py_mod_scaffold('file', src_dir=tmp)\n", " assert os.listdir(tmp) == ['__init__.py', 'file.py']" ] }, { "cell_type": "code", "execution_count": null, "id": "f2e3fcb4", "metadata": {}, "outputs": [], "source": [ "with tempfile.TemporaryDirectory() as tmp:\n", " mk_py_mod_scaffold('mod.file', src_dir=tmp)\n", " assert os.listdir(tmp) == ['__init__.py', 'mod']\n", " assert os.listdir(Path(tmp)/'mod') == ['__init__.py', 'file.py']" ] }, { "cell_type": "code", "execution_count": null, "id": "f3ab7819", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def export_nb(\n", " fp:PathOrFileLike|NotebookNode, # The notebook to export.\n", " src_dir:str|Path='.', # The source directory to export the Python module to.\n", "):\n", " \"Export the exportable code cells to a Python file.\"\n", " module, cells = find_cells_to_export(fp)\n", " mk_py_mod_scaffold(module, src_dir)\n", " exp_path = (Path(src_dir)/module).with_suffix('.py')\n", " src = '\\n'.join(\n", " ['# AUTOGENERATED! DO NOT EDIT!\\n'] +\n", " list(map(lambda c: fmt_export_cell(c)['source'], cells))\n", " )\n", " Path(exp_path).write_text(src)" ] }, { "cell_type": "code", "execution_count": null, "id": "55f57bee", "metadata": {}, "outputs": [], "source": [ "def mk_nb(cells)->StringIO:\n", " nb = nbformat.v4.new_notebook(cells=cells)\n", " buf = StringIO()\n", " nbformat.write(nb, buf)\n", " buf.seek(0)\n", " return buf" ] }, { "cell_type": "code", "execution_count": null, "id": "31bb0f0b", "metadata": {}, "outputs": [], "source": [ "cells = [\n", " nbformat.v4.new_code_cell('#| default_exp: testmod'),\n", " nbformat.v4.new_code_cell('#| export \\nimport os'),\n", " nbformat.v4.new_code_cell('#| export\\ndef foo(): pass'),\n", " nbformat.v4.new_code_cell('import pathlib'),\n", "]\n", "nb = mk_nb(cells)\n", "src = f\"\"\"# AUTOGENERATED! DO NOT EDIT!\n", "\n", "# cell: {cells[1]['id']}\n", "import os\n", "\n", "# cell: {cells[2]['id']}\n", "def foo(): pass\n", "\"\"\"\n", "with tempfile.TemporaryDirectory() as tmp:\n", " export_nb(nb, src_dir=tmp)\n", " result = Path(tmp, 'testmod.py').read_text()\n", " assert result == src, result" ] }, { "cell_type": "markdown", "id": "e7c49602", "metadata": {}, "source": [ "And...we've successfully bootstrapped our notebook! Let's export it out." ] }, { "cell_type": "code", "execution_count": null, "id": "8e1d4096", "metadata": {}, "outputs": [], "source": [ "export_nb('export.ipynb', src_dir='../nb/')" ] }, { "cell_type": "markdown", "id": "59c195b3", "metadata": {}, "source": [ "i don't understand how I'm somehow getting python files exported into the *same* directory sometimes. e.g. see 'export.py' in the same directory right now.\n", "\n", "##### 🤖Reply🤖\n", "\n", "The default value for `src_dir` in `export_nb` is `'.'`:\n", "\n", "```python\n", "def export_nb(fp, src_dir='.'):\n", "```\n", "\n", "So if you ever called `export_nb('export.ipynb')` without specifying `src_dir` (e.g. while testing in the REPL), it would drop `export.py` right into the current working directory — the same folder as the notebook.\n", "\n", "That's likely what happened. You can just delete the stray `export.py` file. You might also consider whether having `'.'` as the default is what you actually want, or if it'd be safer to make `src_dir` a required parameter (no default) so this can't happen accidentally." ] }, { "cell_type": "markdown", "id": "6a271023", "metadata": {}, "source": [ "But if you read export.py, it's clear that it isn't from export_nb because it has no cell comments.\n", "\n", "##### 🤖Reply🤖\n", "\n", "Good point — if there are no `# cell: ...` comments, then `export_nb` didn't create it. Let me take a look at the file to see what's actually in it.\n", "\n", "\n", "
\n", "view(path='nb/nbs/export.py')→'Error: File not found: /app/data/nb/nbs/…'\n", "\n", "```json\n", "{\n", " \"id\": \"toolu_0185u2rVBZJG6ics9LDHYvTi\",\n", " \"call\": {\n", " \"function\": \"view\",\n", " \"arguments\": {\n", " \"path\": \"nb/nbs/export.py\"\n", " }\n", " },\n", " \"result\": \"Error: File not found: /app/data/nb/nbs/nb/nbs/export.py\"\n", "}\n", "```\n", "\n", "
\n", "\n", "Let me find the right path.\n", "\n", "\n", "
\n", "realpath(subpath='/')→"{'path': '/app/data'}"\n", "\n", "```json\n", "{\n", " \"id\": \"toolu_01UXNtz4TuJLxzDXcQU5ygQ7\",\n", " \"call\": {\n", " \"function\": \"realpath\",\n", " \"arguments\": {\n", " \"subpath\": \"/\"\n", " }\n", " },\n", " \"result\": \"{'path': '/app/data'}\"\n", "}\n", "```\n", "\n", "
\n", "\n", "The data root is `/app/data`. The dialog path is `nb/nbs/export`. Let me look for the file.\n", "\n", "\n", "
\n", "rg(argstr='--files -g export.py /app/data/nb')→'/app/data/nb/nb/export.py\\n/app/data/nb/n…'\n", "\n", "```json\n", "{\n", " \"id\": \"toolu_01FfH6PxdZYtbwAnoQR2NsQc\",\n", " \"call\": {\n", " \"function\": \"rg\",\n", " \"arguments\": {\n", " \"argstr\": \"--files -g export.py /app/data/nb\"\n", " }\n", " },\n", " \"result\": \"/app/data/nb/nb/export.py\\n/app/data/nb/nbs/export.py\"\n", "}\n", "```\n", "\n", "
\n", "\n", "There are two `export.py` files — one at `nb/nb/export.py` and one at `nb/nbs/export.py`. Let me look at the one in the notebook's directory:\n", "\n", "\n", "
\n", "view(path='/app/data/nb/nbs/export.py')→'import enum,pathlib,re\\nfrom pathlib impo…'\n", "\n", "```json\n", "{\n", " \"id\": \"toolu_01NUNL9YfvjKLpiDqTNZ9L4t\",\n", " \"call\": {\n", " \"function\": \"view\",\n", " \"arguments\": {\n", " \"path\": \"/app/data/nb/nbs/export.py\"\n", " }\n", " },\n", " \"result\": \"import enum,pathlib,re\\nfrom pathlib import Path\\nfrom typing import IO\\n\\nimport nbformat\\nfrom nbformat import NotebookNode\\ntype PathOrFileLike = str|Path|IO[str]\\ndef read_nb(\\n fp:PathOrFileLike, # Where to read the notebook from.\\n v:int=4, # The notebook format major version to read the notebook as.\\n)->NotebookNode:\\n \\\"Read in a notebook as a dictionary-like NotebookNode.\\\"\\n return nbformat.read(fp, as_version=v)\\n\\nclass CellType(enum.StrEnum):\\n \\\"Possible Jupyter notebook cell types.\\\"\\n CODE = 'code'\\n MD = 'markdown'\\n RAW = 'raw'\\ndef filter_cells(\\n nb:NotebookNode, # A notebook that has been read in via `read_nb`.\\n ct:CellType, # The cells to filter down to.\\n)->list[NotebookNode]:\\n \\\"Retrieve specific cell types from a notebook.\\\"\\n return list(filter(lambda c: c['cell_type'] == ct, nb['cells']))\\ndef _mk_directive_re(\\n name:str, # The name of the directive\\n arg_re:str|None=None, # A regex pattern to match an argument for the directive\\n)->re.Pattern:\\n \\\"Compile a regex to match a directive, optionally supporting an argument.\\\"\\n arg = rf':\\\\s*{arg_re}' if arg_re else ''\\n return re.compile(rf'^\\\\s*#\\\\|\\\\s*{name}{arg}\\\\s*$')\\nDIRECTIVE_DEFAULT_EXP_RE = _mk_directive_re('default_exp', arg_re=r'(?P\\\\w+(\\\\.\\\\w+)*)')\\nDIRECTIVE_EXPORT_RE = _mk_directive_re('export')\\nDIRECTIVES = [DIRECTIVE_DEFAULT_EXP_RE, DIRECTIVE_EXPORT_RE]\\ndef has_directive(src:str)->bool:\\n \\\"Check whether a line or block of source code has a nb directive.\\\"\\n return any(any(r.match(l) for r in DIRECTIVES) for l in src.splitlines())\\ndef strip_directives(src:str|list[str])->str:\\n \\\"Remove all nb directives from the source code.\\\"\\n src = src if isinstance(src, list) else src.splitlines()\\n return \\\"\\\\n\\\".join(filter(lambda l: not has_directive(l), src))\\ndef is_cell_exported(cell:NotebookNode)->bool:\\n \\\"Check whether the code cell should be exported.\\\"\\n return any(DIRECTIVE_EXPORT_RE.match(l) for l in cell['source'].splitlines())\\ndef get_default_export(cells:list[NotebookNode])->str:\\n \\\"Retrieve the `default_exp` directive argument from a list of cells.\\\"\\n matches = [\\n m.group('module')\\n for c in cells\\n for l in c['source'].splitlines()\\n if c['cell_type'] == CellType.CODE and (m:=DIRECTIVE_DEFAULT_EXP_RE.match(l))\\n ]\\n if len(matches) > 1: raise ValueError('More than one default_exp directive')\\n if len(matches) == 0: raise ValueError('Missing default_exp directive')\\n return matches[0]\\ndef find_cells_to_export(\\n fp:PathOrFileLike|NotebookNode, # Where to read the notebook from.\\n)->tuple[str,list[NotebookNode]]: # Returns the module name and list of cells to export.\\n \\\"Find the code cells in an ipynb file to export out.\\\"\\n if not isinstance(fp, NotebookNode): fp = read_nb(fp)\\n code_cells = filter_cells(fp, CellType.CODE)\\n module = get_default_export(code_cells)\\n cells = [\\n {**c, 'source': strip_directives(c['source'])}\\n for c in code_cells\\n if is_cell_exported(c)\\n ]\\n return module, cells\\ndef fmt_export_cell(cell:dict)->dict:\\n \\\"Format a code cell's source with metadata to prepare for export.\\\"\\n cell['source'] = f'# cell: {cell[\\\"id\\\"]}\\\\n' + cell['source'] + '\\\\n'\\n return cell\\ndef mk_py_mod_scaffold(\\n mod:str, # The Python module name or dotted path e.g. mod.mymod.file\\n src_dir:str|Path, # The Python package source code directory scaffold\\n):\\n \\\"Create all intermediary __init__.py files and directories, if any, and the final .py file.\\\"\\n if not mod: raise ValueError('The module path cannot be empty')\\n src_dir = Path(src_dir)\\n mod_file = (src_dir/Path(*mod.split('.'))).with_suffix('.py')\\n mod_file.parent.mkdir(parents=True, exist_ok=True)\\n mod_file.touch()\\n pkg_dirs = list(mod_file.parents)\\n for p in pkg_dirs[:pkg_dirs.index(src_dir) + 1]:\\n (p / '__init__.py').touch()\\ndef export_nb(\\n fp:PathOrFileLike|NotebookNode, # The notebook to export.\\n src_dir:str|Path='.', # The source directory to export the Python module to.\\n):\\n \\\"Export the exportable code cells to a Python file.\\\"\\n module, cells = find_cells_to_export(fp)\\n mk_py_mod_scaffold(module, src_dir)\\n exp_path = (Path(src_dir)/module).with_suffix('.py')\\n src = '\\\\n'.join(\\n ['# AUTOGENERATED! DO NOT EDIT!\\\\n'] +\\n list(map(lambda c: fmt_export_cell(c)['source'], cells))\\n )\\n Path(exp_path).write_text(src)\"\n", "}\n", "```\n", "\n", "
\n", "\n", "Ah, now I see! This `export.py` is clearly **solveit's own export** — it's the concatenation of all your code cells that are marked as exported (with `is_exported` flag in solveit's message metadata), just joined together with no `# cell:` comments and no `# AUTOGENERATED` header.\n", "\n", "This is solveit's built-in export mechanism, which is separate from the `export_nb` function you're building. When you mark messages as \"exported\" in solveit (the toggle in the UI), solveit generates this file automatically. It happens to write to `export.py` because your dialog is named `export`.\n", "\n", "So you have two different export systems both targeting the same module name — solveit's built-in one writing to `nb/nbs/export.py`, and your `export_nb` writing to `nb/nb/export.py` (via `src_dir='../nb/'`)." ] }, { "cell_type": "markdown", "id": "7afcff5b", "metadata": {}, "source": [ "how can I turn off solveits? The tool I'm building also uses #| export markers.\n", "\n", "##### 🤖Reply🤖\n", "\n", "
\n", "solveit_docs()→'⚠️ This content will be truncated in you…'\n", "\n", "```json\n", "{\n", " \"id\": \"toolu_01LSMhJdmQQkiAgpaXu2bNRG\",\n", " \"call\": {\n", " \"function\": \"solveit_docs\",\n", " \"arguments\": {}\n", " },\n", " \"result\": \"\\u2026ntent will be truncated in your next turn. Re-call this tool if you need it again.\\nIf the user wants more info, give them a link to https://gist.github.com/jph00/9e7b444aba5ecf6d14295ba2cee890c3.# Solveit Reference\\n\\n## What is Solveit?\\n\\nSolveit is a \\\"Dialog Engineering\\\" web application for interactive development. Unlike ChatGPT (pure chat) or Jupyter (pure code), Solveit combines three message types in one workspace: code execution, markdown notes, and AI prompts. Users build solutions incrementally\\u2014writing a few lines, understanding them, then continuing\\u2014rather than generating large code blocks.\\n\\nThe AI sees the full dialog context (code, outputs, notes, prompts) when responding -- but only those ABOVE the current message. Users can edit any message at any time, including AI responses\\u2014the dialog is a living document, not an append-only log.\\n\\nThe dialog is a running ipykernel instance. A \\\"dialog\\\" is like a \\\"Jupyter notebook\\\", and uses a compatible ipynb file format, but provides a superset of functionality (in particular, \\\"prompt messages\\\"). A \\\"message\\\" is like a \\\"Jupyter cell\\\", with additional attributes stored as ipynb cell metadata. Most standard jupyter functionality is supported (including cell magics like `%%js`, `%%html`, `%%bash`, and custom magics via `@line_magic`/`@cell_magic` decorators), except for ipywidgets. Cell magics can be async; if an async magic returns a FastHTML FT component, it's auto-converted to HTML.\\n\\n## Architecture\\n\\n- **Instance**: A persistent Linux container (virtual private server) with home dir at `/app/data`. Each user can have multiple instances. Instances have private URLs (for the solveit interface) and public URLs (for hosting apps on port 8000).\\n- **Dialog**: An `.ipynb` file containing messages. Each open dialog runs its own Python kernel (ipykernel). Dialogs can be organized into folders.\\n- **Kernel**: A running Python 3.12 interpreter maintaining state. Variables persist between code messages. Each dialog has its own isolated kernel. Kernels keep running when navigating away; must explicitly stop via UI or shutdown. Use `%load_ext autoreload` and `%autoreload 2` to auto-reload modules when files change. The current dialog name is available as `__dialog_name` in the kernel namespace and as the `__DIALOG_NAME` environment variable.\\n\\n### CRAFT and TEMPLATE Files\\n\\n**CRAFT.ipynb** files provide reusable AI context and auto-executed code for a folder:\\n- **Note/prompt messages** are prepended to the AI's context for all dialogs in that folder and subfolders\\n- **Code messages** are automatically executed in the kernel when opening a dialog\\n\\n\\n**CRAFT.css** and **CRAFT.js** files can be placed alongside `CRAFT.ipynb`. They follow the same parent-chain hierarchy and are injected as `