Masked Language Model Loss Code

def lm_loss(network_output, target, mask): """ batch_size, seq_len, num_classes = network_output.shape flat_output = network_output.view(-1, num_classes) flat_target = target.view(batch_size * seq_len) flat_mask = mask.view(batch_size * seq_len) per_token_loss = F.cross_entropy(flat_output, flat_target, reduce=False) return th.sum(per_token_loss * flat_mask) / th.sum(flat_mask)

A screenshot of Python code defining a masked language-model loss function. The function flattens network outputs, targets, and a mask, calculates per-token cross-entropy, and averages the loss over unmasked tokens. It is shared as a programming meme about machine-learning code and tensor-shape manipulation.

• @profunctor_io / 1115 2018-07-26