I noted that Scipy must be storing some intermediate arrays when doing matrix multiplication. I assume this can be helpful in some cases, but it is a pain sometimes. Consider the following example:
from scipy.sparse import coo_matrix n = 100000000000 row = np.array((0, 0)) col = np.array((0, n-1)) data = np.array((1, 1)) A = coo_matrix((data, (row, col)), shape=(2, n))
Yes, this is a very large matrix. However it has only two nonzero values. The result of
B = A.dot(A.T) can be evaluated by hand since it has only one nonzero value. The matrix
B is such that
B(0, 0) = 2 and zero elsewhere. When I try to run this I get Memory Error, you can see the last lines below:
402 col = self.col.astype(idx_dtype, copy=False) 403 --> 404 indptr = np.empty(M + 1, dtype=idx_dtype) 405 indices = np.empty_like(col, dtype=idx_dtype) 406 data = np.empty_like(self.data, dtype=upcast(self.dtype)) MemoryError:
The output is a 2 x 2 matrix, so it doesn’t matter if it is dense or not. What the program is trying to do to cause this failure at such simple problem? How can I fix this behavior?