python – Scipy sparse memory explosion with simple matrix multiplication


I noted that Scipy must be storing some intermediate arrays when doing matrix multiplication. I assume this can be helpful in some cases, but it is a pain sometimes. Consider the following example:

from scipy.sparse import coo_matrix
n = 100000000000
row  = np.array((0, 0))
col  = np.array((0, n-1))
data = np.array((1, 1))
A = coo_matrix((data, (row, col)), shape=(2, n))

Yes, this is a very large matrix. However it has only two nonzero values. The result of B = A.dot(A.T) can be evaluated by hand since it has only one nonzero value. The matrix B is such that B(0, 0) = 2 and zero elsewhere. When I try to run this I get Memory Error, you can see the last lines below:

    402             col = self.col.astype(idx_dtype, copy=False)
    403 
--> 404             indptr = np.empty(M + 1, dtype=idx_dtype)
    405             indices = np.empty_like(col, dtype=idx_dtype)
    406             data = np.empty_like(self.data, dtype=upcast(self.dtype))

MemoryError: 

The output is a 2 x 2 matrix, so it doesn’t matter if it is dense or not. What the program is trying to do to cause this failure at such simple problem? How can I fix this behavior?

Thank you.