{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "aed9daa9-f05d-4e25-b9a5-a74d0211ef6f", "metadata": {}, "outputs": [], "source": [ "%%capture\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": null, "id": "596de171-5567-43c1-98ce-66eccc28bbe9", "metadata": {}, "outputs": [], "source": [ "#| default_exp: core" ] }, { "cell_type": "code", "execution_count": null, "id": "6a2cc7f8-6c65-4a7c-890c-3cf9d1cd314e", "metadata": {}, "outputs": [], "source": [ "#| export\n", "import enum,inspect,re\n", "from typing import Callable\n", "\n", "import nbformat\n", "from nbformat import NotebookNode" ] }, { "cell_type": "code", "execution_count": null, "id": "bbaf531f", "metadata": {}, "outputs": [], "source": [ "import contextlib\n", "from pathlib import Path\n", "from tempfile import TemporaryDirectory\n", "from nb import test" ] }, { "cell_type": "markdown", "id": "8485c16a", "metadata": {}, "source": [ "# Core\n", "> Fundamental Utilities" ] }, { "cell_type": "markdown", "id": "bdc4df64", "metadata": {}, "source": [ "## Notebooks" ] }, { "cell_type": "markdown", "id": "3f99ca39", "metadata": {}, "source": [ "### Reading Notebooks" ] }, { "cell_type": "markdown", "id": "268db6ac", "metadata": {}, "source": [ "To read in `ipynb` files, we can use a library provided Jupyter called `nbformat`." ] }, { "cell_type": "code", "execution_count": null, "id": "d9d0a18c", "metadata": {}, "outputs": [], "source": [ "#| export\n", "type PathOrFileLike = str|Path|IO[str]" ] }, { "cell_type": "code", "execution_count": null, "id": "ca71fdf5", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def read_nb(\n", " fp:PathOrFileLike, # Where to read the notebook from.\n", " v:int=4, # The notebook format major version to read the notebook as.\n", ")->NotebookNode:\n", " \"Read in a notebook as a dictionary-like NotebookNode.\"\n", " return nbformat.read(fp, as_version=v)" ] }, { "cell_type": "code", "execution_count": null, "id": "b4ca48d4", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook()\n", "with TemporaryDirectory() as tmp, contextlib.chdir(tmp):\n", " with open('nb.ipynb', mode='w+') as fp: nbformat.write(nb, fp)\n", " for fp in ['nb.ipynb', Path('nb.ipynb')]:\n", " nb = read_nb(fp)\n", " test.equal(list(nb.keys()), ['cells', 'metadata', 'nbformat', 'nbformat_minor'])\n", " test.equal(type(nb), NotebookNode)" ] }, { "cell_type": "markdown", "id": "a9abdd44", "metadata": {}, "source": [ "We can see that the top-level format of a notebook is:\n", "- `cells`: A list of objects for each cell in the notebook.\n", "- `metadata`: Free-form notebook level metadata used by tools and editors.\n", "- `nbformat`: Notebook format major version\n", "- `nbformat_minor`: Notebook format minor version" ] }, { "cell_type": "markdown", "id": "772d813e", "metadata": {}, "source": [ "### Filtering Notebooks" ] }, { "cell_type": "markdown", "id": "8cf77bcc", "metadata": {}, "source": [ "Within cells, there are generally 3 types of cells:\n", "- `code`\n", "- `markdown`\n", "- `raw`" ] }, { "cell_type": "code", "execution_count": null, "id": "2ffd7bcf", "metadata": {}, "outputs": [], "source": [ "#| export\n", "class CellType(enum.StrEnum):\n", " \"Possible Jupyter notebook cell types.\"\n", " CODE = 'code'\n", " MD = 'markdown'\n", " RAW = 'raw'" ] }, { "cell_type": "code", "execution_count": null, "id": "a5b24f3a", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def filter_cells(\n", " nb:NotebookNode, # A notebook that has been read in via `read_nb`.\n", " ct:CellType, # The cells to filter down to.\n", ")->list[NotebookNode]:\n", " \"Retrieve specific cell types from a notebook.\"\n", " return list(filter(lambda c: c['cell_type'] == ct, nb['cells']))" ] }, { "cell_type": "code", "execution_count": null, "id": "807f0146", "metadata": {}, "outputs": [], "source": [ "nb = nbformat.v4.new_notebook(cells=[\n", " nbformat.v4.new_code_cell(),\n", " nbformat.v4.new_markdown_cell(),\n", " nbformat.v4.new_raw_cell(),\n", "])" ] }, { "cell_type": "code", "execution_count": null, "id": "5b89c5d8", "metadata": {}, "outputs": [], "source": [ "code_cell = filter_cells(nb, CellType.CODE)[0]\n", "test.equal(list(code_cell.keys()), ['id', 'cell_type', 'metadata', 'execution_count', 'source', 'outputs'])\n", "test.equal(code_cell['cell_type'], CellType.CODE)" ] }, { "cell_type": "code", "execution_count": null, "id": "170144a7", "metadata": {}, "outputs": [], "source": [ "md_cell = filter_cells(nb, CellType.MD)[0]\n", "test.equal(list(md_cell.keys()), ['id', 'cell_type', 'source', 'metadata'])\n", "test.equal(md_cell['cell_type'], CellType.MD)" ] }, { "cell_type": "code", "execution_count": null, "id": "edbe00dd", "metadata": {}, "outputs": [], "source": [ "raw_cell = filter_cells(nb, CellType.RAW)[0]\n", "test.equal(list(raw_cell.keys()), ['id', 'cell_type', 'source', 'metadata'])\n", "test.equal(raw_cell['cell_type'], CellType.RAW)" ] }, { "cell_type": "markdown", "id": "558e2214", "metadata": {}, "source": [ "Within each cell object, they will all have:\n", "- `id`: A unique identifier for that cell.\n", "- `cell_type`: One of the 3 cell type names.\n", "- `source`: The content of the cell, whether it is prose or code.\n", "- `metadata`: Free-form metadata for that cell used by various tools/editors.\n", "\n", "Additionally, `code` cell types will have:\n", "- `outputs`: The stdout/stderr of the cell after the code is executed.\n", "- `execution_count`: The number of times the cell was run." ] }, { "cell_type": "markdown", "id": "e4d56d07", "metadata": {}, "source": [ "### Directives" ] }, { "cell_type": "markdown", "id": "97162699", "metadata": {}, "source": [ "A regular expression ran against each line in a cell should allow us to find each directive we support. Since we'll need to re-create this regression expression for a few directives, we'll create a small helper." ] }, { "cell_type": "code", "execution_count": null, "id": "40433e19", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def mk_directive_re(\n", " name:str, # The name of the directive\n", " arg_re:str|None=None, # A regex pattern to match an argument for the directive\n", ")->re.Pattern:\n", " \"Compile a regex to match a directive, optionally supporting an argument.\"\n", " arg = rf':\\s*{arg_re}' if arg_re else ''\n", " return re.compile(rf'^\\s*#\\|\\s*{name}{arg}\\s*$')" ] }, { "cell_type": "code", "execution_count": null, "id": "ac907b7c", "metadata": {}, "outputs": [], "source": [ "_TEST_DIRECTIVE_RE = mk_directive_re('test')" ] }, { "cell_type": "code", "execution_count": null, "id": "40f7de66", "metadata": {}, "outputs": [], "source": [ "test_dir_match = [\n", " ' #| test \\t \\n', # Lots of whitespace around each token\n", " '#|test', # No whitespace around each token\n", "]\n", "for line in test_dir_match:\n", " test.is_true(_TEST_DIRECTIVE_RE.match(line))" ] }, { "cell_type": "code", "execution_count": null, "id": "f115789b", "metadata": {}, "outputs": [], "source": [ "test_dir_no_match = [\n", " '| test', # Missing #\n", " '# test', # Missing |\n", " '#| testt', # Misspelled directive\n", " '#| test test', # Extra word after directive\n", "]\n", "for line in test_dir_no_match:\n", " test.is_false(_TEST_DIRECTIVE_RE.match(line))" ] }, { "cell_type": "code", "execution_count": null, "id": "3f58f798", "metadata": {}, "outputs": [], "source": [ "_TEST_ARG_DIRECTIVE_RE = mk_directive_re('test', arg_re=r'\\w')" ] }, { "cell_type": "code", "execution_count": null, "id": "2bbc9f93", "metadata": {}, "outputs": [], "source": [ "test_dir_arg_match = [\n", " ' #| test: \\t a \\n', # Lots of whitespace around each token\n", " '#|test:b', # No whitespace around each token\n", "]\n", "for line in test_dir_arg_match:\n", " test.is_true(_TEST_ARG_DIRECTIVE_RE.match(line))" ] }, { "cell_type": "code", "execution_count": null, "id": "c994255e", "metadata": {}, "outputs": [], "source": [ "test_arg_dir_no_match = [\n", " '#| test a', # Missing :\n", " '#| test: ', # Missing arg\n", " '#| test: aa', # Invalid arg (should be only 1 letter)\n", "]\n", "for line in test_dir_no_match + test_arg_dir_no_match:\n", " test.is_false(_TEST_ARG_DIRECTIVE_RE.match(line))" ] }, { "cell_type": "code", "execution_count": null, "id": "535b3ce2", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def has_directive(src:str)->bool:\n", " \"Check whether a line or block of source code has a nb directive.\"\n", " from nb.export import DIRECTIVES as EXPORT_DIRECTIVES\n", " return any(any(r.match(l) for r in EXPORT_DIRECTIVES) for l in src.splitlines())" ] }, { "cell_type": "code", "execution_count": null, "id": "b42fe082", "metadata": {}, "outputs": [], "source": [ "test.is_true(has_directive('#| default_exp: bootstrap'))\n", "test.is_true(has_directive('#| export'))\n", "test.is_true(has_directive('max(1,20)\\n#| export'))\n", "test.is_false(has_directive('# comment\\nmax(1, 20)\\n'))" ] }, { "cell_type": "code", "execution_count": null, "id": "d02bf0bc", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def strip_directives(src:str|list[str])->str:\n", " \"Remove all nb directives from the source code.\"\n", " src = src if isinstance(src, list) else src.splitlines()\n", " return \"\\n\".join(filter(lambda l: not has_directive(l), src))" ] }, { "cell_type": "code", "execution_count": null, "id": "9eca0c79", "metadata": {}, "outputs": [], "source": [ "test.equal(strip_directives('#| export\\na=5'), 'a=5')\n", "test.equal(strip_directives('#| default_exp: bootstrap\\na=5'), 'a=5')\n", "test.equal(strip_directives(['a=10', 'a+=10', '#|export']), 'a=10\\na+=10')" ] }, { "cell_type": "markdown", "id": "2b6822dd", "metadata": {}, "source": [ "## Function Utilities" ] }, { "cell_type": "code", "execution_count": null, "id": "c4875cc0-2d87-4c87-8789-7dc950be06c6", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def callable_name(c: Callable) -> str:\n", " \"Get the name of the callable `c`.\"\n", " return getattr(c, '__name__', c.__class__.__name__)" ] }, { "cell_type": "markdown", "id": "406d605d", "metadata": {}, "source": [ "This works for any function:" ] }, { "cell_type": "code", "execution_count": null, "id": "0e9c39c5", "metadata": {}, "outputs": [], "source": [ "assert callable_name(max) == \"max\"" ] }, { "cell_type": "markdown", "id": "d66b2d42", "metadata": {}, "source": [ "And for any classes:" ] }, { "cell_type": "code", "execution_count": null, "id": "056cb776", "metadata": {}, "outputs": [], "source": [ "class _A: pass\n", "assert callable_name(_A) == \"_A\"" ] }, { "cell_type": "code", "execution_count": null, "id": "f2c70dd3", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def pubattr(o):\n", " \"Retrieve the public attributes of the object `o`.\"\n", " return [a for a in dir(o) if not a.startswith('_')]" ] }, { "cell_type": "markdown", "id": "53cb2bd5", "metadata": {}, "source": [ "`pubattr` will hide any attribute on an object if it starts with an underscore." ] }, { "cell_type": "code", "execution_count": null, "id": "ac30af03", "metadata": {}, "outputs": [], "source": [ "_A.print = print\n", "_A._private_print = print\n", "_A.__python_internal__ = print\n", "assert pubattr(_A) == [\"print\"]" ] }, { "cell_type": "markdown", "id": "f892f4f1", "metadata": {}, "source": [ "## Class Definitions" ] }, { "cell_type": "markdown", "id": "4008335c", "metadata": {}, "source": [ "Iterative notebook-driven development makes class definitions tricky. Typically, you'd write a clas definition in one monolithic block, which leaves no room for prose or incremental execution. Inspired by the wonderful [fastcore](https://github.com/AnswerDotAI/fastcore) library, we'll add `patch` and `patch_to` function to enable piecemeal class definition via monkeypatching." ] }, { "cell_type": "code", "execution_count": null, "id": "78d61a07", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def patch_to(\n", " o:object, # The object to patch.\n", " cls_method:bool=False, # Patch the decorated function to `o` as a classmethod\n", " static_method:bool=False, # Patch the decorated function to `o` as a staticmethod\n", " prop:bool=False, # Patch the decorated function to `o` as a property\n", ")->Callable:\n", " \"Decorator that takes in a object and attaches the decorated function onto it.\"\n", " def inner(fn:Callable)->Callable:\n", " nm = callable_name(fn)\n", " if cls_method: fn = classmethod(fn)\n", " if static_method: fn = staticmethod(fn)\n", " if prop: fn = property(fn)\n", " setattr(o, nm, fn)\n", " return fn\n", " return inner\n" ] }, { "cell_type": "markdown", "id": "0d4bb2e6", "metadata": {}, "source": [ "Let's try it out. We'll create a dummy class named `_T`." ] }, { "cell_type": "code", "execution_count": null, "id": "c67a3c29", "metadata": {}, "outputs": [], "source": [ "class _T: pass" ] }, { "cell_type": "markdown", "id": "b7d0a482", "metadata": {}, "source": [ "Now, we'll attach on a function to `_T` in a separate code cell." ] }, { "cell_type": "code", "execution_count": null, "id": "42580f8b", "metadata": {}, "outputs": [], "source": [ "@patch_to(_T)\n", "def say_hello(self: _T, name: str): return f'Hello, {name}'" ] }, { "cell_type": "markdown", "id": "0edc152b", "metadata": {}, "source": [ "`_T` should have a `say_hello` function on it. Let's confirm." ] }, { "cell_type": "code", "execution_count": null, "id": "404bd72f", "metadata": {}, "outputs": [], "source": [ "_t = _T()\n", "test.equal(_t.say_hello('eugene'), 'Hello, eugene')" ] }, { "cell_type": "markdown", "id": "b9c102e7", "metadata": {}, "source": [ "We can also do this with class methods:" ] }, { "cell_type": "code", "execution_count": null, "id": "f5eca4e3", "metadata": {}, "outputs": [], "source": [ "@patch_to(_T, cls_method=True)\n", "def say_good_morning(cls:_T): return 'Good morning'" ] }, { "cell_type": "code", "execution_count": null, "id": "3a52ddb4", "metadata": {}, "outputs": [], "source": [ "test.equal(_T.say_good_morning(), 'Good morning')" ] }, { "cell_type": "markdown", "id": "29565467", "metadata": {}, "source": [ "And with static methods:" ] }, { "cell_type": "code", "execution_count": null, "id": "da2e0279", "metadata": {}, "outputs": [], "source": [ "@patch_to(_T, static_method=True)\n", "def say_good_evening(): return 'Good evening'" ] }, { "cell_type": "code", "execution_count": null, "id": "d3802615", "metadata": {}, "outputs": [], "source": [ "test.equal(_T.say_good_evening(), 'Good evening')" ] }, { "cell_type": "markdown", "id": "6e694762", "metadata": {}, "source": [ "And properties:" ] }, { "cell_type": "code", "execution_count": null, "id": "be5a1f11", "metadata": {}, "outputs": [], "source": [ "@patch_to(_T, prop=True)\n", "def hi(self:_T): return 'hi'" ] }, { "cell_type": "code", "execution_count": null, "id": "d637ab8e", "metadata": {}, "outputs": [], "source": [ "test.equal(_T().hi, 'hi')" ] }, { "cell_type": "markdown", "id": "4acd774c", "metadata": {}, "source": [ "That's neat and all but notice that `self: _T` and `@patch_to(_T)` are both two different ways to provide the same sort of information (attach this method onto `_T`).\n", "\n", "What if we could look at the type hint to determine which object to patch?\n", "\n", "We already have `patch_to`, so let's create a `patch` decorator. All we'll need to do inside `patch` is find the object to patch and call `patch_to`. If we assume the object we're patching will always be called `self` i.e. patching a class, then this should be easy-peasy." ] }, { "cell_type": "code", "execution_count": null, "id": "4eb650ab", "metadata": {}, "outputs": [], "source": [ "#| export\n", "def patch(\n", " fn:Callable|None=None, # The function to patch onto the object specified via a `self` or `cls` type hint.\n", " prop:bool=False, # Whether to patch `fn` onto the `self` type hint as a property.\n", ")->Callable: # The patched `fn`\n", " \"\"\"Decorator to patch an object using type hints on `fn`.\n", "\n", " `fn` must have either `self` or `cls` arguments present with a type hint.\n", " - `self`: Created as either an instance method or property if `prop=True`.\n", " - `cls`: Created as a class method\n", "\n", " If `self` and `cls` is missing or `self` is missing with `prop=True`,\n", " a ValueError is raised.\n", " \"\"\"\n", " def inner(fn:Callable)->Callable:\n", " anno = inspect.get_annotations(fn)\n", " if 'self' not in anno and prop: raise ValueError('Must have `self` argument with prop=True')\n", " if 'self' in anno: return patch_to(anno['self'], prop=prop)(fn)\n", " if 'cls' in anno: return patch_to(anno['cls'], cls_method=True)(fn)\n", " raise ValueError('Must have either `self` or `cls` argument when using patch')\n", " if fn is not None: return inner(fn) # @patch case, as opposed to @patch()\n", " return inner" ] }, { "cell_type": "markdown", "id": "13272b5e", "metadata": {}, "source": [ "The one caveat is we can't easily support static methods because we naturally may have no arguments to investigate the type hints of to know which object to patch with static methods." ] }, { "cell_type": "markdown", "id": "97378af3", "metadata": {}, "source": [ "Let's see if we can try `patch` out now with our dummy `_T` class." ] }, { "cell_type": "code", "execution_count": null, "id": "601e27f2", "metadata": {}, "outputs": [], "source": [ "@patch\n", "def say_goodbye(self:_T, name): return f'Goodbye, {name}'" ] }, { "cell_type": "code", "execution_count": null, "id": "6aa1ffbc", "metadata": {}, "outputs": [], "source": [ "test.equal(_T().say_goodbye('eugene'), 'Goodbye, eugene')" ] }, { "cell_type": "code", "execution_count": null, "id": "b5d14f35", "metadata": {}, "outputs": [], "source": [ "@patch\n", "def say_bye(cls:_T): return 'bye'" ] }, { "cell_type": "code", "execution_count": null, "id": "f3ce8fe7", "metadata": {}, "outputs": [], "source": [ "test.equal(_T.say_bye(), 'bye')" ] }, { "cell_type": "code", "execution_count": null, "id": "0c59c003", "metadata": {}, "outputs": [], "source": [ "@patch(prop=True)\n", "def bye(self:_T): return 'bye'" ] }, { "cell_type": "code", "execution_count": null, "id": "5fedf001", "metadata": {}, "outputs": [], "source": [ "test.equal(_T().bye, 'bye')" ] }, { "cell_type": "code", "execution_count": null, "id": "12b62cf8-2193-4f03-b80a-46304d73896e", "metadata": {}, "outputs": [], "source": [ "import nb; nb.export_nb('core.ipynb', src_dir='../nb/')" ] } ], "metadata": {}, "nbformat": 4, "nbformat_minor": 5 }