Tag: index

I am working on an information retrieval project, where I have to process text data of ~ 1.5 GB and create a Dictionary (words, document frequency) and a list of publications (document identification, frequency of terms). According to the teacher, it should take around 10-15 minutes. But my code is working for more than 8 hours now! I tried a smaller data set (~ 35 MB) and it took me 5 hours to process it.

I am a Python newbie and I think it is taking a long time because I have created many Python dictionaries and lists in my code. I tried using the generator, but I'm not sure how to fix it.

file = open(filename, 'rt')
text = file.read()
file.close()

p = r'
.*?
' 
tag = RegexpTokenizer(p)
passage = tag.tokenize(text)
doc_re = re.compile(r"
")

def process_data(docu):
    tokens = RegexpTokenizer(r'w+')    
    lower_tokens = (word.lower() for word in tokens.tokenize(docu)) #convert to lower case  
    table = str.maketrans('','', string.punctuation)
    stripped = (w.translate(table) for w in lower_tokens) #remove punctuation   
    alpha = (word for word in stripped if word.isalpha()) #remove tokens that are not alphabetic    
    stopwordlist = stopwords.words('english')
    stopped = (w for w in alpha if not w in stopwordlist) #remove stopwords

    return stopped

data = {} #dictionary: key = Doc ID, value: word/term
for doc in passage:
    group_docID = doc_re.match(doc)
    docID = group_docID.group(1)
    tokens = process_data(doc)
    data(docID) = list(set(tokens))

vocab = (item for i in data.values() for item in i) #all words in the dataset
total_vocab = list(set(vocab)) #unique word/vocbulary for the whole dataset
total_vocab.sort()
print('Document Size = ', len(data)) #no. of documents
print('Collection Size = ', len(vocab)) #no. of words
print('Vocabulary Size= ', len(total_vocab)) #no. of unique words


inv_index = {} #dictionary: key =word/term, value: (docid, termfrequency)
for x in total_vocab:
    for y, z in data.items():
        if x in z:
            wordfreq = z.count(x)
            inv_index.setdefault(x, ()).append((int(y), wordfreq)) 

flattend = (item for tag in inv_index.values() for item in tag) #((docid, tf))
posting = (item  for tag in flattend for item in tag ) #(docid, tf)

#document frequency for each vocabulary/words
doc_freq=()
for k,v in inv_index.items():
    freq1=len((item for item in v if item))
    doc_freq.append((freq1))

#offset value of each vocabulary/words
offset = ()
offset1=0
for i in range(len(doc_freq)):
    if i>0:
        offset1 =offset1 + (doc_freq(i-1)*2)
    offset.append((offset1))

#create dcitionary of words, document frequency and offset
dictionary = {}
for i in range(len(total_vocab)):
    dictionary(total_vocab(i))=(doc_freq(i),offset(i))

#dictionary of word, inverse document frequency
idf = {}
for i in range(len(dictionary)):
    a = np.log2(len(data)/doc_freq(i))
    idf(total_vocab(i)) = a

with open('dictionary.json', 'w') as f:
    json.dump(dictionary,f)

with open('idf.json', 'w') as f:
    json.dump(idf, f)

binary_file = open('binary_file.txt', 'wb')

for i in range(0, len(posting)):
    binary_int = (posting(i)).to_bytes(4, byteorder = 'big')
    #print(binary_int)
    binary_file.write(binary_int)

binary_file.close()

Could someone help me rewrite this code to make it more computational and time efficient?

There are about 57982 documents like that.
Input File:

Background
Adrenal cortex oncocytic carcinoma (AOC) represents an exceptional 
pathological entity, since only 22 cases have been documented in the 
literature so far.
Case presentation
Our case concerns a 54-year-old man with past medical history of right 
adrenal excision with partial hepatectomy, due to an adrenocortical 
carcinoma. The patient was admitted in our hospital to undergo surgical 
resection of a left lung mass newly detected on chest Computed Tomography 
scan. The histological and immunohistochemical study revealed a metastatic 
AOC. Although the patient was given mitotane orally in adjuvant basis, he 
experienced relapse with multiple metastases in the thorax twice.....

I am trying to tokenize each document by word and store the frequency of each word in a dictionary. Trying to save it in the json file.
Dictionary

word document_frequency offset
medical 2500 3414
research 320 4200

Also, generate an index where each word has a publication list of document ID and frequency of terms

medical (2630932, 20), (2795320, 2), (26350421, 31).... 
research (2783546, 243), (28517364, 310)....

and then save these posts in a binary file:

2630932 20 2795320 2 2635041 31....

with a offset value for each word. So when you load the list of publications from disk, you could use the search function to get the publication of each corresponding word.