Skip to content
GitLab

Bugfix - Generation API fails if the generated sequence is truncated.

Merged Administrator requested to merge punitkoura/bug_fix_distribution into main

Created by: punitkoura

Patch Description

  1. Ensure that we pad the mask properly when calculating the token distribution.
  2. Add a function to decode tokens in utils.py - This will be used internally.

Testing steps Ran an API locally to verify generation works.

curl -k <URL> -H "Authorization: Bearer Punit" -H "Content-Type: application/json" \
> -d '{
> "prompt": [2, 44019, 28049, 13770, 6439, 22824, 2306, 14380, 6804, 5457, 21523, 50118, 50118, 713, 16, 10, 569, 9, 10, 11954, 6439, 3704, 81],
> "temperature": 0.0,
> "max_tokens": 300, "min_tokens": 0,
> "top_p": 1.0, "n": 1, "best_of": 1,
> "echo": true, "logprobs": 1, "seed": 1
> }'