python with pyinstrument seems to show a memory leak

[mattfil]

Hi, I am using Units from python. One thing that I have noticed is that, when I set up a test with pyinstrument @profile (https://github.com/joerick/pyinstrument), the inscances of Unit seems to be leaked. Here a minimal example:

import functools
from collections.abc import Callable
from typing import Any, TypeVar

from memory_profiler import profile
from pyinstrument import Profiler
from units_llnl import Unit

F = TypeVar("F", bound=Callable[..., Any])

PROFILED_FILES: list[str] = []


def profile_function(output_file: str) -> Callable[[F], F]:

    def decorator(func: F) -> F:
        @functools.wraps(func)
        def wrapper(*args: Any, **kwargs: Any) -> Any:
            profiler = Profiler(0.00001, async_mode="enabled")
            profiler.start()
            result = func(*args, **kwargs)  # Run the function
            profiler.stop()
            namefile = f"{output_file}.html"
            profiler.write_html(namefile)
            return result

        return wrapper  # type: ignore

    return decorator


@profile
@profile_function("eval")
def test_silly() -> None:
    class dummy:
        def __init__(self, theunit: str) -> None:
            self.unit = Unit(1, theunit)

    for ii in range(100):
        aa = dummy("m")
        bb = dummy("kg")

For such example i get in the terminal:

nanobind: leaked 2 instances!
nanobind: leaked 1 types!
  leaked type "units_llnl.units_llnl_ext.Unit"
nanobind: leaked 34 functions!
  leaked function "to_dict"
  leaked function "__mul__"
  leaked function "is_equation"
  leaked function "isinf"
  leaked function "__hash__"
  leaked function "__ne__"
  leaked function "__truediv__"
  leaked function "is_valid"
  leaked function "is_exactly_the_same"
  leaked function ""
  leaked function "is_convertible_to"
  ... skipped remainder
nanobind: this is likely caused by a reference counting issue in the binding code.

This actually seems a true leak, here the report from the memory_profiler:

46    263.5 MiB    263.5 MiB           1           @functools.wraps(func)
47                                                 def wrapper(*args: Any, **kwargs: Any) -> Any:
48    263.5 MiB      0.0 MiB           1               profiler = Profiler(0.00001, async_mode="enabled")
49    263.5 MiB      0.0 MiB           1               profiler.start()
50    266.6 MiB      3.1 MiB           1               result = func(*args, **kwargs)  # Run the function
51    266.6 MiB      0.0 MiB           1               profiler.stop()
52    266.6 MiB      0.0 MiB           1               namefile = f"{output_file}.html"
53    267.6 MiB      1.0 MiB           1               profiler.write_html(namefile)
54    267.6 MiB      0.0 MiB           1               return result

Thank you, Mattia

[phlptp]

Thanks, I think I am going to have to understand a lot more about python memory management and how that interacts with nanobind to understand what is going on here.

[phlptp]

If you have any suggestions of things that could be done differently they would be welcome

[thorstenhater]

After recent experiences with nb:

1. Check whether this happens during normal operation. If not, instrumentation or internal errors are the issue.
2. Is a simple import units sufficient? If so, Default argument values

https://nanobind.readthedocs.io/en/latest/refleaks.html#sources-of-reference-leaks