Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • M metaseq
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 95
    • Issues 95
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 41
    • Merge requests 41
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Administrator
  • metaseq
  • Merge requests
  • !411

Run evals during training

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Administrator requested to merge evalhook into main Oct 14, 2022
  • Overview 21
  • Commits 14
  • Pipelines 0
  • Changes 2

Created by: ruanslv

Patch Description Added a few config options so we can kick off an arbitrary evals command during training. This allows us to keep logic in OSS land without adding dependencies to our metaseq-internal evals code.

Instead of having the two command requirements, we could also add logic for them in metaseq directly, but they are cloud env specific, so it would mean more of our azure internal logic getting added to OSS land.

Instead, for now internally we can keep the logic in the eval scripts.

Testing steps Using our baseline sweeps setting the new params. Example: --eval_command = f""" python metaseq_internal/scripts/eval/schedule_jobs_few_shot_opt_evaluation.py -o {output_path} -t opt_eval_tasks --model-name {args.prefix} --model-template bf16_sharded_config --slurm-partition zetta """

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: evalhook