INNER CODE UNIT · Python

__getitem__

Koziev/NLP_Datasets · Conversations/Data/run_chitchat_query.py:37

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        sample = self.examples[idx]

        if self.transform:
            sample = self.transform(sample)

        return sample


if __name__ == '__main__':
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model_dir = os.path.expanduser('~/polygon/chatbot/tmp/rugpt_chitchat')

    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    tokenizer.add_special_tokens({'bos_token': '<s>', 'eos_token': '</s>', 'pad_token': '<pad>'})

View source record →