python-performance-optimization

upstream

Profile and optimize Python code using cProfile, memory profilers, and performance best practices. Use when debugging slow Python code, optimizing bottlenecks, or improving application performance.

ID: python-performance-optimization

Volver

Python Performance Optimization

Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.

When to Use This Skill

Identifying performance bottlenecks in Python applications
Reducing application latency and response times
Optimizing CPU-intensive operations
Reducing memory consumption and memory leaks
Improving database query performance
Optimizing I/O operations
Speeding up data processing pipelines
Implementing high-performance algorithms
Profiling production applications

Core Concepts

1. Profiling Types

CPU Profiling: Identify time-consuming functions
Memory Profiling: Track memory allocation and leaks
Line Profiling: Profile at line-by-line granularity
Call Graph: Visualize function call relationships

2. Performance Metrics

Execution Time: How long operations take
Memory Usage: Peak and average memory consumption
CPU Utilization: Processor usage patterns
I/O Wait: Time spent on I/O operations

3. Optimization Strategies

Algorithmic: Better algorithms and data structures
Implementation: More efficient code patterns
Parallelization: Multi-threading/processing
Caching: Avoid redundant computation
Native Extensions: C/Rust for critical paths

Quick Start

Basic Timing

``python


import time


def measure_time():
    """Simple timing measurement."""
    start = time.time()


    # Your code here
    result = sum(range(1000000))


    elapsed = time.time() - start
    print(f"Execution time: {elapsed:.4f} seconds")
    return result


Better: use timeit for accurate measurements
import timeit


execution_time = timeit.timeit(
    "sum(range(1000000))",
    number=100
)
print(f"Average time: {execution_time/100:.6f} seconds")




Profiling Tools


Pattern 1: cProfile - CPU Profiling

`python


import cProfile
import pstats
from pstats import SortKey


def slow_function():
    """Function to profile."""
    total = 0
    for i in range(1000000):
        total += i
    return total


def another_function():
    """Another function."""
    return [i2 for i in range(100000)]



def main():
    """Main function to profile."""
    result1 = slow_function()
    result2 = another_function()
    return result1, result2


Profile the code
if __name__ == "__main__":
    profiler = cProfile.Profile()
    profiler.enable()


    main()


    profiler.disable()


    # Print stats
    stats = pstats.Stats(profiler)
    stats.sort_stats(SortKey.CUMULATIVE)
    stats.print_stats(10)  # Top 10 functions


    # Save to file for later analysis
    stats.dump_stats("profile_output.prof")

Command-line profiling:**

`bash


Profile a script
python -m cProfile -o output.prof script.py


View results
python -m pstats output.prof
In pstats:
sort cumtime
stats 10




Pattern 2: line_profiler - Line-by-Line Profiling

`python


Install: pip install line-profiler


Add @profile decorator (line_profiler provides this)
@profile
def process_data(data):
    """Process data with line profiling."""
    result = []
    for item in data:
        processed = item * 2
        result.append(processed)
    return result


Run with:
kernprof -l -v script.py




Manual line profiling:

`python


from line_profiler import LineProfiler


def process_data(data):
    """Function to profile."""
    result = []
    for item in data:
        processed = item * 2
        result.append(processed)
    return result


if __name__ == "__main__":
    lp = LineProfiler()
    lp.add_function(process_data)


    data = list(range(100000))


    lp_wrapper = lp(process_data)
    lp_wrapper(data)


    lp.print_stats()




Pattern 3: memory_profiler - Memory Usage

`python


Install: pip install memory-profiler


from memory_profiler import profile


@profile
def memory_intensive():
    """Function that uses lots of memory."""
    # Create large list
    big_list = [i for i in range(1000000)]


    # Create large dict
    big_dict = {i: i2 for i in range(100000)}



    # Process data
    result = sum(big_list)


    return result


if __name__ == "__main__":
    memory_intensive()


Run with:
python -m memory_profiler script.py

`
Pattern 4: py-spy - Production Profiling
`bash
Install: pip install py-spy Profile a running Python process py-spy top --pid 12345 Generate flamegraph py-spy record -o profile.svg --pid 12345 Profile a script py-spy record -o profile.svg -- python script.py Dump current call stack py-spy dump --pid 12345
`
Optimization Patterns Pattern 5: List Comprehensions vs Loops
`python
import timeit Slow: Traditional loop def slow_squares(n): """Create list of squares using loop.""" result = [] for i in range(n):

        result.append(i2)
    return result


Fast: List comprehension
def fast_squares(n):
    """Create list of squares using comprehension."""
    return [i2 for i in range(n)]



Benchmark
n = 100000


slow_time = timeit.timeit(lambda: slow_squares(n), number=100)
fast_time = timeit.timeit(lambda: fast_squares(n), number=100)


print(f"Loop: {slow_time:.4f}s")
print(f"Comprehension: {fast_time:.4f}s")
print(f"Speedup: {slow_time/fast_time:.2f}x")


Even faster for simple operations: map
def faster_squares(n):
    """Use map for even better performance."""
    return list(map(lambda x: x2, range(n)))




Pattern 6: Generator Expressions for Memory

`python


import sys


def list_approach():
    """Memory-intensive list."""
    data = [i2 for i in range(1000000)]

    return sum(data)


def generator_approach():
    """Memory-efficient generator."""
    data = (i2 for i in range(1000000))
    return sum(data)


Memory comparison
list_data = [i for i in range(1000000)]
gen_data = (i for i in range(1000000))


print(f"List size: {sys.getsizeof(list_data)} bytes")
print(f"Generator size: {sys.getsizeof(gen_data)} bytes")


Generators use constant memory regardless of size




Pattern 7: String Concatenation

`python


import timeit


def slow_concat(items):
    """Slow string concatenation."""
    result = ""
    for item in items:
        result += str(item)
    return result


def fast_concat(items):
    """Fast string concatenation with join."""
    return "".join(str(item) for item in items)


def faster_concat(items):
    """Even faster with list."""
    parts = [str(item) for item in items]
    return "".join(parts)


items = list(range(10000))


Benchmark
slow = timeit.timeit(lambda: slow_concat(items), number=100)
fast = timeit.timeit(lambda: fast_concat(items), number=100)
faster = timeit.timeit(lambda: faster_concat(items), number=100)


print(f"Concatenation (+): {slow:.4f}s")
print(f"Join (generator): {fast:.4f}s")
print(f"Join (list): {faster:.4f}s")




Pattern 8: Dictionary Lookups vs List Searches

`python


import timeit


Create test data
size = 10000
items = list(range(size))
lookup_dict = {i: i for i in range(size)}


def list_search(items, target):
    """O(n) search in list."""
    return target in items


def dict_search(lookup_dict, target):
    """O(1) search in dict."""
    return target in lookup_dict


target = size - 1  # Worst case for list


Benchmark
list_time = timeit.timeit(
    lambda: list_search(items, target),
    number=1000
)
dict_time = timeit.timeit(
    lambda: dict_search(lookup_dict, target),
    number=1000
)


print(f"List search: {list_time:.6f}s")
print(f"Dict search: {dict_time:.6f}s")
print(f"Speedup: {list_time/dict_time:.0f}x")




Pattern 9: Local Variable Access

`python


import timeit


Global variable (slow)
GLOBAL_VALUE = 100


def use_global():
    """Access global variable."""
    total = 0
    for i in range(10000):
        total += GLOBAL_VALUE
    return total


def use_local():
    """Use local variable."""
    local_value = 100
    total = 0
    for i in range(10000):
        total += local_value
    return total


Local is faster
global_time = timeit.timeit(use_global, number=1000)
local_time = timeit.timeit(use_local, number=1000)


print(f"Global access: {global_time:.4f}s")
print(f"Local access: {local_time:.4f}s")
print(f"Speedup: {global_time/local_time:.2f}x")




Pattern 10: Function Call Overhead

`python


import timeit


def calculate_inline():
    """Inline calculation."""
    total = 0
    for i in range(10000):
        total += i * 2 + 1
    return total


def helper_function(x):
    """Helper function."""
    return x * 2 + 1


def calculate_with_function():
    """Calculation with function calls."""
    total = 0
    for i in range(10000):
        total += helper_function(i)
    return total


Inline is faster due to no call overhead
inline_time = timeit.timeit(calculate_inline, number=1000)
function_time = timeit.timeit(calculate_with_function, number=1000)


print(f"Inline: {inline_time:.4f}s")
print(f"Function calls: {function_time:.4f}s")




Advanced Optimization


Pattern 11: NumPy for Numerical Operations

`python


import timeit
import numpy as np


def python_sum(n):
    """Sum using pure Python."""
    return sum(range(n))


def numpy_sum(n):
    """Sum using NumPy."""
    return np.arange(n).sum()


n = 1000000


python_time = timeit.timeit(lambda: python_sum(n), number=100)
numpy_time = timeit.timeit(lambda: numpy_sum(n), number=100)


print(f"Python: {python_time:.4f}s")
print(f"NumPy: {numpy_time:.4f}s")
print(f"Speedup: {python_time/numpy_time:.2f}x")


Vectorized operations
def python_multiply():
    """Element-wise multiplication in Python."""
    a = list(range(100000))
    b = list(range(100000))
    return [x * y for x, y in zip(a, b)]


def numpy_multiply():
    """Vectorized multiplication in NumPy."""
    a = np.arange(100000)
    b = np.arange(100000)
    return a * b


py_time = timeit.timeit(python_multiply, number=100)
np_time = timeit.timeit(numpy_multiply, number=100)


print(f"\nPython multiply: {py_time:.4f}s")
print(f"NumPy multiply: {np_time:.4f}s")
print(f"Speedup: {py_time/np_time:.2f}x")




Pattern 12: Caching with functools.lru_cache

`python


from functools import lru_cache
import timeit


def fibonacci_slow(n):
    """Recursive fibonacci without caching."""
    if n < 2:
        return n
    return fibonacci_slow(n-1) + fibonacci_slow(n-2)


@lru_cache(maxsize=None)
def fibonacci_fast(n):
    """Recursive fibonacci with caching."""
    if n < 2:
        return n
    return fibonacci_fast(n-1) + fibonacci_fast(n-2)


Massive speedup for recursive algorithms
n = 30


slow_time = timeit.timeit(lambda: fibonacci_slow(n), number=1)
fast_time = timeit.timeit(lambda: fibonacci_fast(n), number=1000)


print(f"Without cache (1 run): {slow_time:.4f}s")
print(f"With cache (1000 runs): {fast_time:.4f}s")


Cache info
print(f"Cache info: {fibonacci_fast.cache_info()}")




Pattern 13: Using __slots__ for Memory

`python


import sys


class RegularClass:
    """Regular class with __dict__."""
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z


class SlottedClass:
    """Class with __slots__ for memory efficiency."""
    __slots__ = ['x', 'y', 'z']


    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z


Memory comparison
regular = RegularClass(1, 2, 3)
slotted = SlottedClass(1, 2, 3)


print(f"Regular class size: {sys.getsizeof(regular)} bytes")
print(f"Slotted class size: {sys.getsizeof(slotted)} bytes")


Significant savings with many instances
regular_objects = [RegularClass(i, i+1, i+2) for i in range(10000)]
slotted_objects = [SlottedClass(i, i+1, i+2) for i in range(10000)]


print(f"\nMemory for 10000 regular objects: ~{sys.getsizeof(regular) * 10000} bytes")
print(f"Memory for 10000 slotted objects: ~{sys.getsizeof(slotted) * 10000} bytes")




Pattern 14: Multiprocessing for CPU-Bound Tasks

`python


import multiprocessing as mp
import time


def cpu_intensive_task(n):
    """CPU-intensive calculation."""
    return sum(i**2 for i in range(n))


def sequential_processing():
    """Process tasks sequentially."""
    start = time.time()
    results = [cpu_intensive_task(1000000) for _ in range(4)]
    elapsed = time.time() - start
    return elapsed, results


def parallel_processing():
    """Process tasks in parallel."""
    start = time.time()
    with mp.Pool(processes=4) as pool:
        results = pool.map(cpu_intensive_task, [1000000] * 4)
    elapsed = time.time() - start
    return elapsed, results


if __name__ == "__main__":
    seq_time, seq_results = sequential_processing()
    par_time, par_results = parallel_processing()


    print(f"Sequential: {seq_time:.2f}s")
    print(f"Parallel: {par_time:.2f}s")
    print(f"Speedup: {seq_time/par_time:.2f}x")




Pattern 15: Async I/O for I/O-Bound Tasks

`python


import asyncio
import aiohttp
import time
import requests


urls = [
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/1",
]


def synchronous_requests():
    """Synchronous HTTP requests."""
    start = time.time()
    results = []
    for url in urls:
        response = requests.get(url)
        results.append(response.status_code)
    elapsed = time.time() - start
    return elapsed, results


async def async_fetch(session, url):
    """Async HTTP request."""
    async with session.get(url) as response:
        return response.status


async def asynchronous_requests():
    """Asynchronous HTTP requests."""
    start = time.time()
    async with aiohttp.ClientSession() as session:
        tasks = [async_fetch(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
    elapsed = time.time() - start
    return elapsed, results


Async is much faster for I/O-bound work
sync_time, sync_results = synchronous_requests()
async_time, async_results = asyncio.run(asynchronous_requests())


print(f"Synchronous: {sync_time:.2f}s")
print(f"Asynchronous: {async_time:.2f}s")
print(f"Speedup: {sync_time/async_time:.2f}x")




Database Optimization


Pattern 16: Batch Database Operations

`python


import sqlite3
import time


def create_db():
    """Create test database."""
    conn = sqlite3.connect(":memory:")
    conn.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)")
    return conn


def slow_inserts(conn, count):
    """Insert records one at a time."""
    start = time.time()
    cursor = conn.cursor()
    for i in range(count):
        cursor.execute("INSERT INTO users (name) VALUES (?)", (f"User {i}",))
        conn.commit()  # Commit each insert
    elapsed = time.time() - start
    return elapsed


def fast_inserts(conn, count):
    """Batch insert with single commit."""
    start = time.time()
    cursor = conn.cursor()
    data = [(f"User {i}",) for i in range(count)]
    cursor.executemany("INSERT INTO users (name) VALUES (?)", data)
    conn.commit()  # Single commit
    elapsed = time.time() - start
    return elapsed


Benchmark
conn1 = create_db()
slow_time = slow_inserts(conn1, 1000)


conn2 = create_db()
fast_time = fast_inserts(conn2, 1000)


print(f"Individual inserts: {slow_time:.4f}s")
print(f"Batch insert: {fast_time:.4f}s")
print(f"Speedup: {slow_time/fast_time:.2f}x")




Pattern 17: Query Optimization

`python


Use indexes for frequently queried columns
"""
-- Slow: No index
SELECT * FROM users WHERE email = 'user@example.com';


-- Fast: With index
CREATE INDEX idx_users_email ON users(email);
SELECT * FROM users WHERE email = 'user@example.com';
"""


Use query planning
import sqlite3


conn = sqlite3.connect("example.db")
cursor = conn.cursor()


Analyze query performance
cursor.execute("EXPLAIN QUERY PLAN SELECT * FROM users WHERE email = ?", ("test@example.com",))
print(cursor.fetchall())


Use SELECT only needed columns
Slow: SELECT *
Fast: SELECT id, name




Memory Optimization


Pattern 18: Detecting Memory Leaks

`python


import tracemalloc
import gc


def memory_leak_example():
    """Example that leaks memory."""
    leaked_objects = []


    for i in range(100000):
        # Objects added but never removed
        leaked_objects.append([i] * 100)


    # In real code, this would be an unintended reference


def track_memory_usage():
    """Track memory allocations."""
    tracemalloc.start()


    # Take snapshot before
    snapshot1 = tracemalloc.take_snapshot()


    # Run code
    memory_leak_example()


    # Take snapshot after
    snapshot2 = tracemalloc.take_snapshot()


    # Compare
    top_stats = snapshot2.compare_to(snapshot1, 'lineno')


    print("Top 10 memory allocations:")
    for stat in top_stats[:10]:
        print(stat)


    tracemalloc.stop()


Monitor memory
track_memory_usage()


Force garbage collection
gc.collect()




Pattern 19: Iterators vs Lists

`python


import sys


def process_file_list(filename):
    """Load entire file into memory."""
    with open(filename) as f:
        lines = f.readlines()  # Loads all lines
        return sum(1 for line in lines if line.strip())


def process_file_iterator(filename):
    """Process file line by line."""
    with open(filename) as f:
        return sum(1 for line in f if line.strip())


Iterator uses constant memory
List loads entire file into memory




Pattern 20: Weakref for Caches

`python


import weakref


class CachedResource:
    """Resource that can be garbage collected."""
    def __init__(self, data):
        self.data = data


Regular cache prevents garbage collection
regular_cache = {}


def get_resource_regular(key):
    """Get resource from regular cache."""
    if key not in regular_cache:
        regular_cache[key] = CachedResource(f"Data for {key}")
    return regular_cache[key]


Weak reference cache allows garbage collection
weak_cache = weakref.WeakValueDictionary()


def get_resource_weak(key):
    """Get resource from weak cache."""
    resource = weak_cache.get(key)
    if resource is None:
        resource = CachedResource(f"Data for {key}")
        weak_cache[key] = resource
    return resource


When no strong references exist, objects can be GC'd




Benchmarking Tools


Custom Benchmark Decorator

`python


import time
from functools import wraps


def benchmark(func):
    """Decorator to benchmark function execution."""
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, kwargs)

        elapsed = time.perf_counter() - start
        print(f"{func.__name__} took {elapsed:.6f} seconds")
        return result
    return wrapper


@benchmark
def slow_function():
    """Function to benchmark."""
    time.sleep(0.5)
    return sum(range(1000000))


result = slow_function()

`
Performance Testing with pytest-benchmark
`python
Install: pip install pytest-benchmark def test_list_comprehension(benchmark): """Benchmark list comprehension."""

    result = benchmark(lambda: [i2 for i in range(10000)])
    assert len(result) == 10000


def test_map_function(benchmark):
    """Benchmark map function."""
    result = benchmark(lambda: list(map(lambda x: x2, range(10000))))

    assert len(result) == 10000


Run with: pytest test_performance.py --benchmark-compare

``

Best Practices

1. Profile before optimizing - Measure to find real bottlenecks

2. Focus on hot paths - Optimize code that runs most frequently

3. Use appropriate data structures - Dict for lookups, set for membership

4. Avoid premature optimization - Clarity first, then optimize

5. Use built-in functions - They're implemented in C

6. Cache expensive computations - Use lru_cache

7. Batch I/O operations - Reduce system calls

8. Use generators for large datasets

9. Consider NumPy for numerical operations

10. Profile production code - Use py-spy for live systems

Common Pitfalls

Optimizing without profiling

Using global variables unnecessarily

Not using appropriate data structures

Creating unnecessary copies of data

Not using connection pooling for databases

Ignoring algorithmic complexity

Over-optimizing rare code paths

Not considering memory usage

Resources

cProfile: Built-in CPU profiler

memory_profiler: Memory usage profiling

line_profiler: Line-by-line profiling

py-spy: Sampling profiler for production

NumPy: High-performance numerical computing

Cython: Compile Python to C

PyPy**: Alternative Python interpreter with JIT

Performance Checklist

[ ] Profiled code to identify bottlenecks
[ ] Used appropriate data structures
[ ] Implemented caching where beneficial
[ ] Optimized database queries
[ ] Used generators for large datasets
[ ] Considered multiprocessing for CPU-bound tasks
[ ] Used async I/O for I/O-bound tasks
[ ] Minimized function call overhead in hot loops
[ ] Checked for memory leaks
[ ] Benchmarked before and after optimization

Informacion

Estado	Activo
Origen	upstream
Usos	3
Ultimo uso	23/01 16:57
Actualizado	09/01/2026 02:14

Archivo origen

Z:\Repositorios\core\upstream\plugins\python-development\skills\python-performance-optimization\SKILL.md