Python For Loop Slows Due To Large List

So currently I have a for loop, which causes the python program to die with the program saying 'Killed'. It slows down around 6000 items in, with the program slowly dying at around

Solution 1:

You're right, it's probably because the list becomes large: python list are contiguous spaces in memory. Each time you append to the list, python looks if there is a spot at the next position, and if not he relocates the whole array somewhere where there is enough room. The bigger your array, the more python has a to relocate.

One way around would be to create an array of the right size beforehand.

EDIT: Just to make sure it was clear, I made up an example to illustrate my point. I've made 2 functions. The first one appends the stringified index (to make it bigger) to a list at each iteration, and the other just fills a numpy array:

import numpy as np
import matplotlib.pyplot as plt
fromtime import time

def test_bigList(N):
    L = []
    times = np.zeros(N,dtype=np.float32)

    for i in range(N):
        t0 = time()
        L.append(str(i))
        times[i] = time()-t0

    return times

def test_bigList_numpy(N):
    L = np.empty(N,dtype="<U32")
    times = np.zeros(N,dtype=np.float32)

    for i in range(N):
        t0 = time()
        L[i] = str(i)
        times[i] = time()-t0
    return times

N = int(1e7)
res1 = test_bigList(N)
res2 = test_bigList_numpy(N)

plt.plot(res1,label="list")
plt.plot(res2,label="numpy array")
plt.xlabel("Iteration")
plt.ylabel("Running time")
plt.legend()
plt.title("Evolution of iteration time with the size of an array")
plt.show()

I get the following result:

You can see on the figure that for the list case, you have regularly some peaks (probably due to relocation), and they seem to increase with the size of the list. This example is with short appended strings, but the bigger the string, the more you will see this effect.

If it does not do the trick, then it might be linked to the database itself, but I can't help you without knowing the specifics of the database.