2024-02-26 10:10:59,410 INFO    StreamThr :5808 [internal.py:wandb_internal():86] W&B internal server running at pid: 5808, started at: 2024-02-26 10:10:59.405118
2024-02-26 10:10:59,432 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status
2024-02-26 10:11:01,960 INFO    WriterThread:5808 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/run-z4vsrt4l.wandb
2024-02-26 10:11:01,966 DEBUG   SenderThread:5808 [sender.py:send():382] send: header
2024-02-26 10:11:01,976 DEBUG   SenderThread:5808 [sender.py:send():382] send: run
2024-02-26 10:11:02,447 INFO    SenderThread:5808 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files
2024-02-26 10:11:02,447 INFO    SenderThread:5808 [sender.py:_start_run_threads():1136] run started: z4vsrt4l with start time 1708942259.407921
2024-02-26 10:11:02,462 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: check_version
2024-02-26 10:11:02,463 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: check_version
2024-02-26 10:11:02,575 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: run_start
2024-02-26 10:11:02,585 DEBUG   HandlerThread:5808 [system_info.py:__init__():27] System info init
2024-02-26 10:11:02,585 DEBUG   HandlerThread:5808 [system_info.py:__init__():42] System info init done
2024-02-26 10:11:02,585 INFO    HandlerThread:5808 [system_monitor.py:start():194] Starting system monitor
2024-02-26 10:11:02,585 INFO    SystemMonitor:5808 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-02-26 10:11:02,586 INFO    HandlerThread:5808 [system_monitor.py:probe():214] Collecting system info
2024-02-26 10:11:02,586 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started cpu monitoring
2024-02-26 10:11:02,587 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started disk monitoring
2024-02-26 10:11:02,588 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started gpu monitoring
2024-02-26 10:11:02,589 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started memory monitoring
2024-02-26 10:11:02,590 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started network monitoring
2024-02-26 10:11:02,599 DEBUG   HandlerThread:5808 [system_info.py:probe():151] Probing system
2024-02-26 10:11:02,605 DEBUG   HandlerThread:5808 [gitlib.py:_init_repo():56] git repository is invalid
2024-02-26 10:11:02,605 DEBUG   HandlerThread:5808 [system_info.py:probe():199] Probing system done
2024-02-26 10:11:02,605 DEBUG   HandlerThread:5808 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-02-26T10:11:02.599344', 'startedAt': '2024-02-26T10:10:59.351815', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Gemma-7B-Finetuning-Insurance.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1kan-kI251WUZEPoyknzcoQ3vjRSC1xpP', 'host': '30a0ffea74aa', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.148, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.148, 'min': 0.0, 'max': 0.0}, {'current': 2000.148, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.80147171020508}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}}
2024-02-26 10:11:02,605 INFO    HandlerThread:5808 [system_monitor.py:probe():224] Finished collecting system info
2024-02-26 10:11:02,606 INFO    HandlerThread:5808 [system_monitor.py:probe():227] Publishing system info
2024-02-26 10:11:02,626 INFO    HandlerThread:5808 [system_monitor.py:probe():229] Finished publishing system info
2024-02-26 10:11:02,632 DEBUG   SenderThread:5808 [sender.py:send():382] send: files
2024-02-26 10:11:02,633 INFO    SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now
2024-02-26 10:11:02,659 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: python_packages
2024-02-26 10:11:02,659 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: python_packages
2024-02-26 10:11:02,660 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:11:02,661 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:11:02,667 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:11:02,915 DEBUG   SenderThread:5808 [sender.py:send():382] send: telemetry
2024-02-26 10:11:02,915 DEBUG   SenderThread:5808 [sender.py:send():382] send: config
2024-02-26 10:11:02,917 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:11:02,922 DEBUG   SenderThread:5808 [sender.py:send():382] send: telemetry
2024-02-26 10:11:02,923 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:11:02,923 WARNING SenderThread:5808 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen)
2024-02-26 10:11:02,923 DEBUG   SenderThread:5808 [sender.py:send():382] send: telemetry
2024-02-26 10:11:03,260 INFO    wandb-upload_0:5808 [upload_job.py:push():131] Uploaded file /tmp/tmpae1t1xluwandb/ddsashsh-wandb-metadata.json
2024-02-26 10:11:03,452 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/requirements.txt
2024-02-26 10:11:03,453 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-metadata.json
2024-02-26 10:11:04,924 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:09,925 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:14,926 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:17,660 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:11:17,661 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:11:17,661 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:11:20,845 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:25,845 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:30,853 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:31,500 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/config.yaml
2024-02-26 10:11:32,660 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:11:32,661 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:11:32,700 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:11:36,834 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:41,835 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:46,836 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:47,660 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:11:47,661 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:11:47,700 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:11:51,860 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:11:56,861 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:01,862 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:02,590 DEBUG   SystemMonitor:5808 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-02-26 10:12:02,592 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:12:02,660 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:12:02,661 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:12:02,700 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:12:07,820 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:12,821 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:17,660 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:12:17,661 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:12:17,701 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:12:17,888 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:22,889 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:27,895 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:32,593 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:12:32,661 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:12:32,661 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:12:32,701 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:12:33,800 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:38,800 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:43,801 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:47,661 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:12:47,661 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:12:47,701 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:12:48,881 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:53,882 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:12:58,883 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:02,594 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:13:02,660 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:13:02,661 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:13:02,701 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:13:04,840 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:09,840 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:12,172 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history
2024-02-26 10:13:12,175 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:13:12,177 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:13:12,177 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:13:12,177 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:13:12,178 DEBUG   SenderThread:5808 [sender.py:send():382] send: history
2024-02-26 10:13:12,178 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: summary_record
2024-02-26 10:13:12,184 INFO    SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-26 10:13:12,689 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json
2024-02-26 10:13:15,187 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:17,691 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:13:17,692 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:13:17,728 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:13:20,832 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:25,833 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:30,834 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:32,594 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:13:32,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:13:32,674 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:13:32,674 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:13:35,872 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:36,754 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/config.yaml
2024-02-26 10:13:41,174 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:46,175 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:47,672 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:13:47,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:13:47,674 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:13:51,811 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:13:56,812 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:01,813 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:02,595 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:14:02,672 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:14:02,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:14:02,675 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:14:06,866 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:11,867 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:16,868 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:17,672 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:14:17,674 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:14:17,675 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:14:21,919 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:26,919 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:31,923 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:32,596 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:14:32,672 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:14:32,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:14:32,674 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:14:37,907 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:42,908 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:47,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:14:47,674 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:14:47,675 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:14:48,813 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history
2024-02-26 10:14:48,816 DEBUG   SenderThread:5808 [sender.py:send():382] send: history
2024-02-26 10:14:48,817 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: summary_record
2024-02-26 10:14:48,821 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:48,828 INFO    SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-26 10:14:48,916 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json
2024-02-26 10:14:53,839 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:14:58,840 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:02,597 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:15:02,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:15:02,674 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:15:02,676 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:15:03,943 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:08,944 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:13,945 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:17,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:15:17,675 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:15:17,675 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:15:19,896 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:24,897 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:29,899 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:32,597 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:15:32,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:15:32,674 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:15:32,675 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:15:34,904 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:39,907 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:44,908 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:47,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:15:47,674 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:15:47,675 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:15:50,908 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:15:55,912 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:00,913 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:02,598 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:16:02,673 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:16:02,675 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:16:02,675 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:16:05,915 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:10,250 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history
2024-02-26 10:16:10,253 DEBUG   SenderThread:5808 [sender.py:send():382] send: history
2024-02-26 10:16:10,253 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: summary_record
2024-02-26 10:16:10,260 INFO    SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-26 10:16:11,151 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json
2024-02-26 10:16:11,262 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:16,263 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:17,709 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:16:17,718 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:16:17,720 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:16:21,883 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:26,885 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:31,886 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:32,599 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:16:32,709 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:16:32,710 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:16:32,749 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:16:37,850 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:42,851 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:47,709 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:16:47,710 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:16:47,750 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:16:47,944 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:52,945 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:16:57,945 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:02,600 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:17:02,710 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:17:02,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:17:02,751 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:17:03,931 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:08,932 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:13,933 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:17,711 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:17:17,711 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:17:17,751 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:17:19,866 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:24,866 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:29,868 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:32,601 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:17:32,711 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:17:32,711 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:17:32,751 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:17:34,947 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:39,948 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:44,949 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:47,711 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:17:47,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:17:47,751 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:17:50,933 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:17:55,934 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:00,934 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:02,602 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:18:02,711 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:18:02,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:18:02,751 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:18:05,937 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:08,893 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history
2024-02-26 10:18:08,898 DEBUG   SenderThread:5808 [sender.py:send():382] send: history
2024-02-26 10:18:08,898 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: summary_record
2024-02-26 10:18:08,905 INFO    SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-26 10:18:09,454 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json
2024-02-26 10:18:11,909 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:16,908 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:17,711 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:18:17,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:18:17,720 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:18:21,912 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:26,913 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:31,918 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:32,602 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:18:32,711 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:18:32,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:18:32,752 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:18:37,899 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:42,900 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:47,711 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:18:47,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:18:47,752 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:18:48,880 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:53,873 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:18:58,874 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:02,603 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:19:02,712 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:19:02,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:19:02,752 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:19:04,874 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:09,875 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:14,875 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:17,712 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:19:17,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:19:17,752 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:19:19,949 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:24,950 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:29,951 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:32,604 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:19:32,712 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:19:32,712 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:19:32,752 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:19:35,901 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:40,902 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:45,903 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:47,712 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:19:47,713 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:19:47,752 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:19:50,950 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:19:55,951 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:00,464 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history
2024-02-26 10:20:00,470 DEBUG   SenderThread:5808 [sender.py:send():382] send: history
2024-02-26 10:20:00,470 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: summary_record
2024-02-26 10:20:00,477 INFO    SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-26 10:20:00,698 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json
2024-02-26 10:20:01,478 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:02,605 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:20:05,970 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:20:05,970 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:20:06,128 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:20:07,183 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:12,184 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:17,189 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:20,925 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:20:20,928 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:20:20,965 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:20:23,116 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:28,117 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:32,606 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:20:33,607 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:35,925 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:20:35,926 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:20:35,965 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:20:39,079 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:44,080 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:49,081 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:50,925 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:20:50,926 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:20:50,966 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:20:54,169 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:20:59,170 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:02,607 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:21:04,610 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:05,925 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:21:05,926 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:21:05,966 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:21:10,168 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:15,168 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:20,169 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:20,925 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:21:20,926 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:21:20,966 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:21:26,132 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:30,342 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history
2024-02-26 10:21:30,343 DEBUG   SenderThread:5808 [sender.py:send():382] send: history
2024-02-26 10:21:30,344 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: summary_record
2024-02-26 10:21:30,349 INFO    SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-26 10:21:30,906 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json
2024-02-26 10:21:31,351 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:32,608 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:21:36,012 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:21:36,014 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:21:36,014 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:21:37,213 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:38,703 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history
2024-02-26 10:21:38,706 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:21:38,707 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:21:38,707 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:21:38,708 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:21:38,708 DEBUG   SenderThread:5808 [sender.py:send():382] send: metric
2024-02-26 10:21:38,709 DEBUG   SenderThread:5808 [sender.py:send():382] send: history
2024-02-26 10:21:38,710 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: summary_record
2024-02-26 10:21:38,729 INFO    SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
2024-02-26 10:21:38,924 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json
2024-02-26 10:21:42,731 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:47,732 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:50,553 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: pause
2024-02-26 10:21:50,554 INFO    HandlerThread:5808 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-02-26 10:21:50,554 INFO    HandlerThread:5808 [system_monitor.py:finish():203] Stopping system monitor
2024-02-26 10:21:50,557 DEBUG   SystemMonitor:5808 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-02-26 10:21:50,565 DEBUG   SystemMonitor:5808 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-02-26 10:21:50,557 INFO    HandlerThread:5808 [interfaces.py:finish():202] Joined cpu monitor
2024-02-26 10:21:50,567 INFO    HandlerThread:5808 [interfaces.py:finish():202] Joined disk monitor
2024-02-26 10:21:50,576 INFO    HandlerThread:5808 [interfaces.py:finish():202] Joined gpu monitor
2024-02-26 10:21:50,576 INFO    HandlerThread:5808 [interfaces.py:finish():202] Joined memory monitor
2024-02-26 10:21:50,577 INFO    HandlerThread:5808 [interfaces.py:finish():202] Joined network monitor
2024-02-26 10:21:50,578 DEBUG   SenderThread:5808 [sender.py:send():382] send: stats
2024-02-26 10:21:51,012 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages
2024-02-26 10:21:51,013 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status
2024-02-26 10:21:51,014 DEBUG   SenderThread:5808 [sender.py:send_request():409] send_request: stop_status
2024-02-26 10:21:53,235 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:58,240 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report
2024-02-26 10:21:58,560 DEBUG   HandlerThread:5808 [handler.py:handle_request():146] handle_request: resume
2024-02-26 10:21:58,561 INFO    HandlerThread:5808 [handler.py:handle_request_resume():699] starting system metrics thread
2024-02-26 10:21:58,561 INFO    HandlerThread:5808 [system_monitor.py:start():194] Starting system monitor
2024-02-26 10:21:58,561 INFO    SystemMonitor:5808 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-02-26 10:21:58,563 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started cpu monitoring
2024-02-26 10:21:58,567 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started disk monitoring
2024-02-26 10:21:58,568 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started gpu monitoring
2024-02-26 10:21:58,574 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started memory monitoring
2024-02-26 10:21:58,574 INFO    SystemMonitor:5808 [interfaces.py:start():190] Started network monitoring
2024-02-26 10:21:58,975 INFO    Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/config.yaml