ENH: Use line profiler as context manager

[skjerns]

I'm using an IDE with REPL and execute many lines of code individually. Sometimes I would quickly like to assess which lines in some multi-line codes are slowest

How difficult would it be to integrate line_profiler as a context manager, without manual overhead of wrapping the statement in a function? This would be similar to the ipython cell magic, but without the cell

Something along those lines:

with lineprofiler:
    func()
    func2()
    func3()


#>L1 func()   - 0.200s - 20%
#>L2 func2() - 0.450s - 45%
#>L3 func3() - 0.350s - 35%

related #127, it seems like this was possible (?) in Python 2. Is there a reason it is not working anymore in Python 3?

[Erotemic]

I don't think line profiling ever worked as a decorator. That comment is about CProfile. The way this works is by rewriting the code within a function and then replacing that function, so I don't think that can be done on a block by block basis. The way it works in IPython is a bit different because each cell is a single unit which can be rewritten. Doing it with a context manager would require recognizing that a function used the context manager, and then rewriting the function but only enabling the profiler for certain lines. That might be doable, but is not on the roadmap.

We are currently working on a big release with lots of improvements. You will be able to automatically profile every function in a module or package using a command line argument or TOML config file without modifying the code at all.

FWIW, I find wrapping a block in a function, about as cumbersome as wrapping a block in a context manager. If you are just timing the block interactively, you might want to look at this wrapper I wrote: xdev.profile_now. It outputs the profile information as the block is run. I find that useful for profiling small bits of code in an interactive / scripty context.

[skjerns]

thanks for your fast response!

Looking forward to the big release.

I'm usually programming in a scripting context, in which creating functions prevents me from accessing variables outside of the function scope. Especially when using a REPL, i cannot quickly measure timings of a few lines/block while also making use of variables in the workspace.

I've created something similar, albeit really simple, here. It seems to work in most contexts, but not all

https://github.com/skjerns/skjerns-utils/blob/master/stimer/contextprofiler.py

from contextprofiler import ContextProfiler
import time

with ContextProfiler:
    time.sleep(0.2)
    time.sleep(0.3)
    time.sleep(0.5)

#>L1 time.sleep(0.2) - 0.200s - 20%
#>L2 time.sleep(0.3) - 0.302s - 30%
#>L3 time.sleep(0.5) - 0.502s - 50%

the profile_now that you provided might also be of great help! (anything that allows me to profile without running the whole file will do)

[TTsangSC]

Just my two cents.

The main issues with incorporating this feature is that:

• Like @Erotemic pointed out, LineProfiler profiling relies on having prior exposure to the code object (via .add_function()) executed, because we hash each line in a code object to an int so that we can use C++ dicts to quickly access timing data.
  • In pytest-autoprofile I was able to use line_profiler with doctest... but that was because each line/statement/block is separately passed to exec(), and thus I could just monkey-patch that to shim in a compile() and a LineProfiler.add_function().
• LineProfiler.__enter__() and .__exit__() are already a thing and they have different semantics; essentially, this

  prof.add_callable(some_callable)
  prof.add_callable(another_callable)
  with prof:  # Turn on profiling for the added functions
      some_callable()  # Profiled
      another_callable()  # Profiled
      nonprofiled_callable()  # Not profiled
  
  some_callable()  # Not profiled

  is equivalent to

  some_wrapper = prof(some_callable)
  another_wrapper = prof(another_callable)
  some_wrapper()  # `some_callable` profiled
  another_wrapper()  # `another_callable` profiled
  nonprofiled_callable()  # Not profiled
  some_callable()  # Not profiled

The trick that you used to get the calling frame upon entering a context is pretty neat though! Have done something similar for another project. I wonder if it will make sense to incorporate that into a LineProfiler class method like LineProfiler.profile_all_lines(), which then returns an object similar to your _ProfilerImpl but wraps the result in a line_profiler._line_profiler.LineStats object.

[Erotemic]

I'm open to the idea after we release 0.5.0, but we need to decide on an API. The LineProfiler class already has a context meaning, which effectively enables / disables it. That raises the question: is anyone using it like that? Is it useful?

I wonder if a decent starting point is the GlobalProfiler, because if we implemented that both as a callable and as a context manager we could enable syntax like:

from line_profiler import profile

with profile:
    print('profiled')
    print('Can we profile in the global scope?')

@profile
def my_profiled_function1(arg):
     print('profiled')
     return str(arg)

def my_profiled_function2(arg):
    print('not profiled')
    with profile:
         return str(arg)
    print('not profiled')