INNER CODE UNIT · Python
batchify_with_label
ShannonAI/glyce · glyce/models/latticeLSTM/main.py:197
def batchify_with_label(input_batch_list, gpu, volatile_flag=False):
"""
input: list of words, chars and labels, various length. [[words,biwords,chars,gaz, labels],[words,biwords,chars,labels],...]
words: word ids for one sentence. (batch_size, sent_len)
chars: char ids for on sentences, various length. (batch_size, sent_len, each_word_length)
output:
zero padding for word and char, with their batch length
word_seq_tensor: (batch_size, max_sent_len) Variable
word_seq_lengths: (batch_size,1) Tensor
char_seq_tensor: (batch_size*max_sent_len, max_word_len) Variable
char_seq_lengths: (batch_size*max_sent_len,1) Tensor
char_seq_recover: (batch_size*max_sent_len,1) recover char sequence order
label_seq_tensor: (batch_size, max_sent_len)
mask: (batch_size, max_sent_len)
"""
batch_size = len(input_batch_list)
words = [sent[0] for sent in input_batch_list]
biwords = [sent[1] for sent in input_batch_list]