eval opt-175B

[henan991201]

I noticed you evaluated the opt-175B model, how did it convert to a Megatron-Deepspeed checkpoint? I can not find a 175B huggingface transformers checkpoint. Also, I can not successfully convert the opt-66B checkpoint. @thomasw21 Thanks for any reply!

[thomasw21]

Hi, we used ranking evaluation for opt-175b, ie getting the loglikelihood of examples. Typically a classification task would look like:

sample1: "Review: {REVIEW}\nIs this review positive or negative? positive
sample2: "Review: {REVIEW}\nIs this review positive or negative? negative

We compute the likelihood of both samples, and rank them to get accuracy/recall/f1 for example.

Concerning the way we converted Megatron-DeepSpeed checkpoint, we essentially computed the opposite of https://github.com/huggingface/transformers/blob/main/src/transformers/models/bloom/convert_bloom_original_checkpoint_to_pytorch.py typically mapped each parameters to their stages (in the DeepSpeed sense), you can find:

• conversion script: [WIP] OPT experimentations #51
• modification to Meg-DS: [WIP] Hack my way to get OPT running Megatron-DeepSpeed#301