INNER CODE UNIT · Python

cache_all_items

IBM/transition-amr-parser · src/ibm_neural_aligner/main.py:536

    def cache_all_items(self):
        for idx in tqdm(range(len(self.corpus))):
            _ = self.__getitem__(idx)

    def __getitem__(self, idx):
        if idx in self.cached:
            return self.cached[idx]
        amr = self.corpus[idx]

        item = {}

        # text
        tokenizer_output = self.text_tokenizer.tokenize(amr.tokens)
        item['text_original_tokens'] = tokenizer_output['original_input']
        item['text_tokens'] = tokenizer_output['input_ids']

        # amr
        tokenizer_output = self.amr_tokenizer.tokenize(amr)

View source record →