7 tips to Time Python scripts and control Memory & CPU usage

When running a complex Python program that takes quite a long time to execute, you might want to improve its execution time. But how?

First of all, you need the tools to detect the bottlenecks of your code, i.e. which parts take longer to execute. This way, you can concentrate in speeding these parts first.

And also, you should also control the memory and CPU usage, as it can point you towards new portions of code that could be improved.

Therefore, in this post I’ll comment on 7 different Python tools that give you some insight about the execution time of your functions and the Memory and CPU usage.

1. Use a decorator to time your functions

The simpler way to time a function is to define a decorator that measures the elapsed time in running the function, and prints the result:

Then, you have to add this decorator before the function you want to measure, like

For example, let’s measure how long it takes to sort an array of 2000000 random numbers:

If you run your script, you should see something like

2. Using the timeit module

Anther option is to use the timeit module, which gives you an average time measure.

To run it, execute the following command in your terminal:

where timing_functions is the name of your script.

At the end of the output, you should see something like:

indicating that of 4 times running the test (-n 4), and averaging 5 repetitions on each test (-r 5), the best test result was of 2.08 seconds.

If you don’t specify the number of tests or repetitions, it defaults to 10 loops and 5 repetitions.

3. Using the time Unix command

However, both the decorator and the timeit module are based on Python. This is why the unix time utility may be useful, as it is an external Python measure.

To run the time utility type:

which gives the output:

The first line comes from the decorator we defined, and the other three:

  • Real indicates the total time spent executing the script.
  • User indicates the amount of time the CPU spent executing the script
  • Sys indicates the amount of time spent in kernel-level functions.

Note: as defined in wikipedia, the kernel is a computer program that manages input/output requests from software, and translates them into data processing instructions for the central processing unit (CPU) and other electronic components of a computer.

Therefore, the difference between the real time and the sum of user+sys may indicate the time spent waiting for input/output or that the system is busy running other external tasks.

4. Using the cProfile module

If you want to know how much time is spent on each function and method, and how many times each of them is called, you can use the cProfile module:

Now you’ll see a detailed description of how many times each function in your code is called, and it will be sorted by the cumulative time spent on each one (thanks to the -s cumulative option).

cprofile_python

You’ll see that the total amount of time spent on running your script is higher than before. This is the penalty we pay for measuring the time each function takes to execute.

5. Using line_profiler module

The line_profiler module gives you information about the CPU time spent on each line in your code.

This module has to be installed first, with

Next, you need to specify which functions you want to evaluate using the @profile decorator (you don’t need to import it in your file):

Finally, you can obtain a line by line description of the random_sort2 function by typing:

where the -l flag indicates line-by-line and the -v flag indicates verbose output. With this method, we see that the array construction takes about 44% of the computation time, whereas the sort() method takes the remaining 56%.

kernprof_line_profiler

You will also see that due to the time measurements, the script might take longer to execute.

6. Use the memory_profiler module

The memory_profiler module is used to measure memory usage in your code, on a line-by-line basis. However, it can make your code to run much more slower.

Install it with

Also, it is recommended to install the psutil package, so that the memory_profile runs faster:

In a similar way as the line_profiler, use the @profile decorator to mark which functions to track. Next, type:

yes, the previous script takes longer than the 1 or 2 seconds that took before. And if you didn’t install the psutil package, maybe you’re still waiting for the results!

memory_profiler

Looking at the output, note that the memory usage is expressed in terms of MiB, which stand for mebibyte (1MiB = 1.05MB).

7. Using the guppy package

Finally, with this package you’ll be able to track how many objects of each type (str, tuple, dict, etc) are created at each stage in your code.

Install it with

Next, add it in your code as:

And run your code with:

You’ll see something like the following output:

Guppy_Python

By placing the heap at different places in your code, you can study the object creation and deletion in the script flow.

If you want to learn more about speeding your Python code, I recommend you the book High Performance Python: Practical Performant Programming for Humans, september 2014.

Hope it was useful! 🙂

Don’t forget to share it with your friends!

Google+TwitterLinkedInFacebookReddit

Please, add +Marina Mele in your comments. This way I will get a notification email and I will answer you as soon as possible! :-)