devngho/code_edu_classifier-v3-microsoft_codebert-base

이 모델은 microsoft/codebert-base에 classifier를 추가한 모델입니다. HuggingFaceFW/fineweb-edu-classifier의 코드 버전을 목표로, 코드의 교육성 점수를 평가합니다. 학습에는 bigcode/the-stack-dedup에서 추출한 샘플을 Qwen/Qwen2.5-Coder-32B-Instruct로 평가한 devngho/the-stack-llm-annotations-v2 데이터셋이 사용되었습니다.

이 연구는 Google의 TPU Research Cloud (TRC)의 Cloud TPU 제공으로 수행되었습니다. ⚡

상세

• 제작: devngho
• 언어: code
• 라이선스: mit
• 기반 모델: microsoft/codebert-base

학습 상세

• learning_rate: 3e-4 (cosine)
• warmup_ratio: 0.1
• batch_size: 2048(512*4)
• optimizer: adamw(b1=0.9, b2=0.98, eps=1e-8, weight_decay=0.01)
• duration: 4h 41m
• steps: 6080

학습 장비

TPU v4-8

성능

Validation Report:
              precision    recall  f1-score   support

           0       0.80      0.06      0.10        72
           1       0.62      0.40      0.48       835
           2       0.61      0.62      0.61      2722
           3       0.48      0.72      0.58      1891
           4       0.62      0.02      0.05       623
           5       0.00      0.00      0.00         1

    accuracy                           0.55      6144
   macro avg       0.52      0.30      0.30      6144
weighted avg       0.58      0.55      0.52      6144

Confusion Matrix:
[[   4   36   30    2    0    0]
 [   1  330  464   40    0    0]
 [   0  157 1684  881    0    0]
 [   0    5  516 1361    9    0]
 [   0    0   71  537   15    0]
 [   0    0    0    1    0    0]]

3 이상과 미만으로 구분할 때 f1 score는 약 0.72입니다.

devngho/code_edu_classifier-v3-microsoft_codebert-base

This model is microsoft/codebert-base with classfier head. It is designed to evaluate the educational value of codes, similar to the HuggingFaceFW/fineweb-edu-classifier, but focused on code. The training data comes from devngho/the-stack-llm-annotations-v2 dataset, contains samples extracted from bigcode/the-stack-dedup and evaluated using Qwen/Qwen2.5-Coder-32B-Instruct.

This research was supported with Cloud TPUs from Google's TPU Research Cloud (TRC).⚡

• Developed by: devngho
• Language(s): code
• License: mit
• Base model: microsoft/codebert-base

Training detail

• learning_rate: 3e-4 (cosine)
• warmup_ratio: 0.1
• batch_size: 2048(512*4)
• optimizer: adamw(b1=0.9, b2=0.98, eps=1e-8, weight_decay=0.01)
• duration: 4h 41m
• steps: 6080

Training hardware

TPU v4-8

Performance

Validation Report:
              precision    recall  f1-score   support

           0       0.80      0.06      0.10        72
           1       0.62      0.40      0.48       835
           2       0.61      0.62      0.61      2722
           3       0.48      0.72      0.58      1891
           4       0.62      0.02      0.05       623
           5       0.00      0.00      0.00         1

    accuracy                           0.55      6144
   macro avg       0.52      0.30      0.30      6144
weighted avg       0.58      0.55      0.52      6144

Confusion Matrix:
[[   4   36   30    2    0    0]
 [   1  330  464   40    0    0]
 [   0  157 1684  881    0    0]
 [   0    5  516 1361    9    0]
 [   0    0   71  537   15    0]
 [   0    0    0    1    0    0]]

The F1 score is about 0.72 when separating above and below 3.