File size: 21,861 Bytes
9c3de9e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
2022-01-14 22:15:34,709 INFO    MainThread:7834 [internal.py:wandb_internal():87] W&B internal server running at pid: 7834, started at: 2022-01-14 22:15:34.709583
2022-01-14 22:15:34,711 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: check_version
2022-01-14 22:15:34,712 INFO    WriterThread:7834 [datastore.py:open_for_write():77] open: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/run-24dma583.wandb
2022-01-14 22:15:34,712 DEBUG   SenderThread:7834 [sender.py:send():234] send: header
2022-01-14 22:15:34,712 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: check_version
2022-01-14 22:15:34,785 DEBUG   SenderThread:7834 [sender.py:send():234] send: run
2022-01-14 22:15:34,980 INFO    SenderThread:7834 [dir_watcher.py:__init__():169] watching files in: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files
2022-01-14 22:15:34,980 INFO    SenderThread:7834 [sender.py:_start_run_threads():804] run started: 24dma583 with start time 1642198533
2022-01-14 22:15:34,980 DEBUG   SenderThread:7834 [sender.py:send():234] send: summary
2022-01-14 22:15:34,980 INFO    SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 22:15:34,981 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: run_start
2022-01-14 22:15:35,985 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json
2022-01-14 22:15:37,284 DEBUG   HandlerThread:7834 [meta.py:__init__():40] meta init
2022-01-14 22:15:37,284 DEBUG   HandlerThread:7834 [meta.py:__init__():54] meta init done
2022-01-14 22:15:37,284 DEBUG   HandlerThread:7834 [meta.py:probe():214] probe
2022-01-14 22:15:37,286 DEBUG   HandlerThread:7834 [meta.py:_setup_git():204] setup git
2022-01-14 22:15:37,315 DEBUG   HandlerThread:7834 [meta.py:_setup_git():211] setup git done
2022-01-14 22:15:37,315 DEBUG   HandlerThread:7834 [meta.py:_save_code():92] save code
2022-01-14 22:15:37,326 DEBUG   HandlerThread:7834 [meta.py:_save_code():113] save code done
2022-01-14 22:15:37,326 DEBUG   HandlerThread:7834 [meta.py:_save_patches():130] save patches
2022-01-14 22:15:37,985 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code/run_mlm_flax.py
2022-01-14 22:15:37,986 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code
2022-01-14 22:15:39,986 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:15:40,986 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/diff.patch
2022-01-14 22:15:42,987 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:15:45,607 ERROR   HandlerThread:7834 [meta.py:_save_patches():171] Error generating diff: Command '['git', 'diff', '--submodule=diff', 'HEAD']' timed out after 5 seconds
2022-01-14 22:15:45,607 DEBUG   HandlerThread:7834 [meta.py:_save_patches():172] save patches done
2022-01-14 22:15:45,607 DEBUG   HandlerThread:7834 [meta.py:_save_pip():58] save pip
2022-01-14 22:15:45,607 DEBUG   HandlerThread:7834 [meta.py:_save_pip():72] save pip done
2022-01-14 22:15:45,608 DEBUG   HandlerThread:7834 [meta.py:probe():252] probe done
2022-01-14 22:15:45,643 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:15:45,643 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:15:45,786 DEBUG   SenderThread:7834 [sender.py:send():234] send: config
2022-01-14 22:15:45,787 DEBUG   SenderThread:7834 [sender.py:send():234] send: config
2022-01-14 22:15:45,787 DEBUG   SenderThread:7834 [sender.py:send():234] send: config
2022-01-14 22:15:45,787 DEBUG   SenderThread:7834 [sender.py:send():234] send: files
2022-01-14 22:15:45,787 INFO    SenderThread:7834 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-14 22:15:45,789 INFO    SenderThread:7834 [sender.py:_save_file():939] saving file code/run_mlm_flax.py with policy now
2022-01-14 22:15:45,988 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-metadata.json
2022-01-14 22:15:45,989 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt
2022-01-14 22:15:46,312 INFO    Thread-12 :7834 [upload_job.py:push():137] Uploaded file /tmp/tmpxqv1l1fswandb/2juok80v-code/run_mlm_flax.py
2022-01-14 22:15:46,330 INFO    Thread-11 :7834 [upload_job.py:push():137] Uploaded file /tmp/tmpxqv1l1fswandb/xnc44171-wandb-metadata.json
2022-01-14 22:15:50,990 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:15:59,991 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:15:59,991 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:16:05,368 DEBUG   SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:16:05,997 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml
2022-01-14 22:16:15,132 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:16:15,132 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:16:30,272 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:16:30,272 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:16:35,439 DEBUG   SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:16:39,009 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:16:45,408 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:16:45,408 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:17:00,601 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:17:00,601 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:17:05,512 DEBUG   SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:17:15,756 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:17:15,756 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:17:30,970 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:17:30,971 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:17:35,586 DEBUG   SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:17:46,135 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:17:46,136 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:18:01,309 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:18:01,309 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:18:05,663 DEBUG   SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:18:16,458 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:18:16,458 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:18:31,596 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:18:31,597 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:18:35,742 DEBUG   SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:18:46,731 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:18:46,732 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:19:03,067 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:19:03,953 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:03,953 DEBUG   SenderThread:7834 [sender.py:send():234] send: telemetry
2022-01-14 22:19:03,953 DEBUG   SenderThread:7834 [sender.py:send():234] send: exit
2022-01-14 22:19:03,953 INFO    SenderThread:7834 [sender.py:send_exit():366] handling exit code: 1
2022-01-14 22:19:03,954 INFO    SenderThread:7834 [sender.py:send_exit():368] handling runtime: 208
2022-01-14 22:19:03,954 INFO    SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 22:19:03,954 INFO    SenderThread:7834 [sender.py:send_exit():374] send defer
2022-01-14 22:19:03,954 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:03,955 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:03,955 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-14 22:19:03,955 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:03,955 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-14 22:19:03,955 INFO    SenderThread:7834 [sender.py:transition_state():387] send defer: 1
2022-01-14 22:19:03,955 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:03,955 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-14 22:19:04,011 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,011 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-14 22:19:04,011 INFO    SenderThread:7834 [sender.py:transition_state():387] send defer: 2
2022-01-14 22:19:04,011 DEBUG   SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:19:04,012 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:04,012 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-14 22:19:04,012 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,012 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-14 22:19:04,012 INFO    SenderThread:7834 [sender.py:transition_state():387] send defer: 3
2022-01-14 22:19:04,012 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:04,012 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-14 22:19:04,012 DEBUG   SenderThread:7834 [sender.py:send():234] send: summary
2022-01-14 22:19:04,013 INFO    SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 22:19:04,013 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,013 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-14 22:19:04,013 INFO    SenderThread:7834 [sender.py:transition_state():387] send defer: 4
2022-01-14 22:19:04,013 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:04,013 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-14 22:19:04,013 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,013 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-14 22:19:04,057 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:04,068 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:19:04,068 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json
2022-01-14 22:19:04,198 INFO    SenderThread:7834 [sender.py:transition_state():387] send defer: 5
2022-01-14 22:19:04,198 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:04,198 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:04,198 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-14 22:19:04,199 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,199 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-14 22:19:04,199 INFO    SenderThread:7834 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-14 22:19:04,300 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,068 INFO    Thread-8  :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml
2022-01-14 22:19:05,069 INFO    SenderThread:7834 [dir_watcher.py:finish():313] scan: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files
2022-01-14 22:19:05,069 INFO    SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml config.yaml
2022-01-14 22:19:05,069 INFO    SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/diff.patch diff.patch
2022-01-14 22:19:05,069 INFO    SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt requirements.txt
2022-01-14 22:19:05,069 INFO    SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log output.log
2022-01-14 22:19:05,070 INFO    SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json wandb-summary.json
2022-01-14 22:19:05,070 INFO    SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-metadata.json wandb-metadata.json
2022-01-14 22:19:05,072 INFO    SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code/run_mlm_flax.py code/run_mlm_flax.py
2022-01-14 22:19:05,073 INFO    SenderThread:7834 [sender.py:transition_state():387] send defer: 6
2022-01-14 22:19:05,073 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,081 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:05,081 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-14 22:19:05,081 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:05,081 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-14 22:19:05,081 INFO    SenderThread:7834 [file_pusher.py:finish():177] shutting down file pusher
2022-01-14 22:19:05,183 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,183 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,285 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,285 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,387 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,387 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,488 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,489 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,539 INFO    Thread-13 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml
2022-01-14 22:19:05,556 INFO    Thread-15 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:19:05,561 INFO    Thread-14 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt
2022-01-14 22:19:05,590 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,590 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,599 INFO    Thread-16 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json
2022-01-14 22:19:05,692 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,692 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,794 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,794 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,799 INFO    Thread-7  :7834 [sender.py:transition_state():387] send defer: 7
2022-01-14 22:19:05,800 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:05,800 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-14 22:19:05,800 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:05,800 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-14 22:19:05,896 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:06,218 INFO    SenderThread:7834 [sender.py:transition_state():387] send defer: 8
2022-01-14 22:19:06,218 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:06,218 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:06,218 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-14 22:19:06,219 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:06,219 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-14 22:19:06,219 INFO    SenderThread:7834 [sender.py:transition_state():387] send defer: 9
2022-01-14 22:19:06,219 DEBUG   SenderThread:7834 [sender.py:send():234] send: final
2022-01-14 22:19:06,219 DEBUG   SenderThread:7834 [sender.py:send():234] send: footer
2022-01-14 22:19:06,219 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:06,220 INFO    HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-14 22:19:06,220 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:06,220 INFO    SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-14 22:19:06,320 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:06,320 DEBUG   SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:06,320 INFO    SenderThread:7834 [file_pusher.py:join():182] waiting for file pusher
2022-01-14 22:19:06,598 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: get_summary
2022-01-14 22:19:06,618 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-14 22:19:06,619 DEBUG   HandlerThread:7834 [handler.py:handle_request():130] handle_request: shutdown
2022-01-14 22:19:06,619 INFO    HandlerThread:7834 [handler.py:finish():731] shutting down handler
2022-01-14 22:19:07,220 INFO    WriterThread:7834 [datastore.py:close():281] close: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/run-24dma583.wandb
2022-01-14 22:19:07,575 INFO    SenderThread:7834 [sender.py:finish():1070] shutting down sender
2022-01-14 22:19:07,575 INFO    SenderThread:7834 [file_pusher.py:finish():177] shutting down file pusher
2022-01-14 22:19:07,576 INFO    SenderThread:7834 [file_pusher.py:join():182] waiting for file pusher
2022-01-14 22:19:07,578 INFO    MainThread:7834 [internal.py:handle_exit():77] Internal process exited