2024-03-21 01:54:58,388 INFO    StreamThr :7197 [internal.py:wandb_internal():86] W&B internal server running at pid: 7197, started at: 2024-03-21 01:54:58.384167
2024-03-21 01:54:58,394 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status
2024-03-21 01:55:03,409 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:03,409 INFO    WriterThread:7197 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/run-y11xrjxv.wandb
2024-03-21 01:55:03,434 DEBUG   SenderThread:7197 [sender.py:send():378] send: header
2024-03-21 01:55:03,442 DEBUG   SenderThread:7197 [sender.py:send():378] send: run
2024-03-21 01:55:04,065 INFO    SenderThread:7197 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files
2024-03-21 01:55:04,066 INFO    SenderThread:7197 [sender.py:_start_run_threads():1099] run started: y11xrjxv with start time 1710986098.384977
2024-03-21 01:55:04,082 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: check_version
2024-03-21 01:55:04,083 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: check_version
2024-03-21 01:55:04,202 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: run_start
2024-03-21 01:55:04,211 DEBUG   HandlerThread:7197 [system_info.py:__init__():26] System info init
2024-03-21 01:55:04,211 DEBUG   HandlerThread:7197 [system_info.py:__init__():41] System info init done
2024-03-21 01:55:04,211 INFO    HandlerThread:7197 [system_monitor.py:start():194] Starting system monitor
2024-03-21 01:55:04,211 INFO    SystemMonitor:7197 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-21 01:55:04,212 INFO    HandlerThread:7197 [system_monitor.py:probe():214] Collecting system info
2024-03-21 01:55:04,212 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started cpu monitoring
2024-03-21 01:55:04,213 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started disk monitoring
2024-03-21 01:55:04,214 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started gpu monitoring
2024-03-21 01:55:04,215 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started memory monitoring
2024-03-21 01:55:04,215 INFO    SystemMonitor:7197 [interfaces.py:start():190] Started network monitoring
2024-03-21 01:55:04,228 DEBUG   HandlerThread:7197 [system_info.py:probe():150] Probing system
2024-03-21 01:55:04,234 DEBUG   HandlerThread:7197 [gitlib.py:_init_repo():56] git repository is invalid
2024-03-21 01:55:04,234 DEBUG   HandlerThread:7197 [system_info.py:probe():198] Probing system done
2024-03-21 01:55:04,234 DEBUG   HandlerThread:7197 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-21T01:55:04.228731', 'startedAt': '2024-03-21T01:54:58.281206', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Mistral-7B-Finetuning-Insurance.ipynb%EC%9D%98%20%EC%82%AC%EB%B3%B8', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1glmddDRkqHDhfUT8slC17FnAssneTUkr', 'host': 'd72aea89d4b6', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.176, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.176, 'min': 0.0, 'max': 0.0}, {'current': 2000.176, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.871490478515625}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674781799316406}}
2024-03-21 01:55:04,235 INFO    HandlerThread:7197 [system_monitor.py:probe():224] Finished collecting system info
2024-03-21 01:55:04,235 INFO    HandlerThread:7197 [system_monitor.py:probe():227] Publishing system info
2024-03-21 01:55:04,254 INFO    HandlerThread:7197 [system_monitor.py:probe():229] Finished publishing system info
2024-03-21 01:55:04,260 DEBUG   SenderThread:7197 [sender.py:send():378] send: files
2024-03-21 01:55:04,260 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now
2024-03-21 01:55:04,647 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: python_packages
2024-03-21 01:55:04,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:55:04,648 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: python_packages
2024-03-21 01:55:04,649 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:55:04,656 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:55:05,019 DEBUG   SenderThread:7197 [sender.py:send():378] send: telemetry
2024-03-21 01:55:05,020 DEBUG   SenderThread:7197 [sender.py:send():378] send: config
2024-03-21 01:55:05,021 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:55:05,022 DEBUG   SenderThread:7197 [sender.py:send():378] send: telemetry
2024-03-21 01:55:05,022 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:55:05,022 WARNING SenderThread:7197 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen)
2024-03-21 01:55:05,022 DEBUG   SenderThread:7197 [sender.py:send():378] send: telemetry
2024-03-21 01:55:05,070 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-metadata.json
2024-03-21 01:55:05,070 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/requirements.txt
2024-03-21 01:55:05,160 INFO    wandb-upload_0:7197 [upload_job.py:push():131] Uploaded file /tmp/tmpe7x8kdblwandb/qivfhje6-wandb-metadata.json
2024-03-21 01:55:09,023 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:14,024 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:19,025 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:19,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:55:19,648 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:55:19,652 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:55:24,884 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:29,891 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:31,114 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/config.yaml
2024-03-21 01:55:34,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:55:34,648 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:55:34,689 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:55:35,977 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:40,977 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:45,980 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:49,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:55:49,649 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:55:49,689 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:55:50,983 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:55:55,984 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:00,985 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:04,216 DEBUG   SystemMonitor:7197 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-03-21 01:56:04,217 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:56:04,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:56:04,649 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:56:04,689 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:56:06,724 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:11,725 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:16,726 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:19,648 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:56:19,649 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:56:19,689 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:56:21,983 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:26,984 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:31,985 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:34,218 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:56:34,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:56:34,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:56:34,691 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:56:37,948 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:42,949 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:47,949 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:49,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:56:49,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:56:49,691 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:56:52,960 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:56:57,961 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:02,961 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:04,219 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:57:04,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:57:04,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:57:04,691 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:57:07,993 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:12,994 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:17,995 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:19,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:57:19,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:57:19,691 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:57:23,885 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:28,885 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:33,886 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:34,219 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:57:34,650 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:57:34,651 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:57:34,697 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:57:36,218 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history
2024-03-21 01:57:36,221 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:57:36,221 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:57:36,221 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:57:36,222 DEBUG   SenderThread:7197 [sender.py:send():378] send: metric
2024-03-21 01:57:36,224 DEBUG   SenderThread:7197 [sender.py:send():378] send: history
2024-03-21 01:57:36,225 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: summary_record
2024-03-21 01:57:36,230 INFO    SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-21 01:57:36,349 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json
2024-03-21 01:57:39,233 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:57:44,234 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:03,817 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:03,819 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:03,819 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:04,033 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:58:04,103 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:58:04,105 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:58:04,415 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:05,362 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:58:05,506 INFO    Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/config.yaml
2024-03-21 01:58:10,366 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:15,367 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:20,047 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:58:20,049 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:58:20,049 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:58:21,305 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:26,306 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:31,307 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,904 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status
2024-03-21 01:58:53,911 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-21 01:58:53,911 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,911 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:58:53,913 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:58:53,904 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:58:53,914 DEBUG   SenderThread:7197 [sender.py:send_request():405] send_request: stop_status
2024-03-21 01:58:55,052 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:58:58,517 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:00,166 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:03,518 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:04,222 DEBUG   SenderThread:7197 [sender.py:send():378] send: stats
2024-03-21 01:59:05,167 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:09,223 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:10,169 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:14,224 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:15,170 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive
2024-03-21 01:59:19,225 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report
2024-03-21 01:59:20,171 DEBUG   HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive