INNER CODE UNIT · Python
compute_valid_token_num
yxuansu/SimCTG · dialogue_generation/loss_func.py:3
def compute_valid_token_num(valid_len_list):
res = 0
for one_len in valid_len_list:
res += one_len * (one_len - 1)
return res
def build_mask_matrix(seqlen, valid_len_list, prefix_len = 0):
'''
prefix_len: the length of prefix that we do not want to compute CL loss for.
(1) if a sequence of length 4 contains zero padding token (i.e., the valid length is 4),
then the loss padding matrix looks like
[0., 1., 1., 1.],
[1., 0., 1., 1.],
[1., 1., 0., 1.],
[1., 1., 1., 0.]
(2) if a sequence of length 4 contains 1 padding token (i.e., the valid length is 3),