Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • M metaseq
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 95
    • Issues 95
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 41
    • Merge requests 41
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Administrator
  • metaseq
  • Issues
  • #471
Closed
Open
Issue created Oct 31, 2022 by Administrator@rootOwner

Include run id in train.log

Created by: suchenzang

Right now, our json logging spits out something like the following:

2022-10-31 09:47:44 | INFO | train_inner | {"epoch": 2, "actv_norm": "300.911", "pos_norm": "0.36", "tok_norm": "0.878", "emb_norm": "0.002", "docsperex": "7.02", "loss": "3.814", "ppl": "14.07", "wps": "319347", "ups": "0.15", "wpb": "2.09715e+06", "bsz": "1024", "num_updates": "5685", "lr": "9.6531e-05", "gnorm": "0.231", "clip": "0", "train_wall": "7", "cuda_gb_allocated": "18.1", "cuda_gb_reserved": "61.6", "cuda_gb_free": "61.1", "wall": "40428"}

Proposed change:

  • Add a run_id to the log (e.g. "run_id": "8") that reflects what is passed in when configuring runs (default 0).

This will make stitching together runs in the future a lot easier when we do a series of rollbacks / inflight changes.

Assignee
Assign to
Time tracking