Training DBRX-like model

#13

by nguyenthanhdo - opened Mar 31

Mar 31

Hi, i've seen the team mentioned that the code used for training DBRX are optimized versions of Composer, LLM Foundry, MegaBlocks and Streaming but I found it quite challenging to navigate. I want to pretrain a MOE model (DBRX-like architecture), could you please guide me how would I do it with the opensourced versions of those mentioned libs?

GODCRM

Apr 2

This comment has been hidden

GODCRM

Apr 2

This comment has been hidden

eitanturok

Databricks org Apr 16

Try reading this: https://github.com/mosaicml/llm-foundry/tree/main/scripts/train

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment