The details of Q1 implementation 1. __init__ function Four variables tf_tokens, tf_entities, idf_tokens and idf_entities are respectively initialized to none,which is convenient for the use of the following functions.And the following results will verify these four indicators. 2. index_documents function First, traverse the documents, take out the corresponding text, and connect it as a string. Deal with it with Spacy, and take out the corresponding entity and token respectively. The dict variable is used to save the corresponding doc_ID and frequences, and the token is the same. However, restrictions such as is_stop, is_punch and single word need to be added.Index the entity first, because no other factors need to be considered. To index the token, we need to filter the token of stop and punct, and invalidate the token that appears in entity. First find the token TF and entity TF, and then find the corresponding IDF. 3. split_query function First, define a queries array to hold queries corresponding to different splits. First, select the eligible entity in doe. The second step is to name all the combinations of entities. The third step is to eliminate the frequency of query in all combinations and select the eligible entity combinations. Finally, according to the matching entity combination, the corresponding token and query are obtained. 4. max_score_query function According to the query obtained in the previous step, the corresponding token and entity sets are calculated respectively. For the token set, TF IDF of each token is calculated by the corresponding TF and IDF calculation methods, and the accumulation is saved by S2. Similarly, TFIDF corresponding to each entity is calculated, and the accumulation is saved with S1. Finally, S1 and S2 are respectively given corresponding weights and added to S. Get the query with the largest s, and save the corresponding s and query as result.
