Hugging Face's logo

Goekdeniz-Guelmez
/
J.O.S.I.E.v4o

MLX
Safetensors
18 datasets
German
English
Mixture of Experts
multimodal
vision
audio
endtoend
j.o.s.i.e.
Model card Files Community
1
Isaak Carter Augustus commited on
Commit
cdf0a4a
•
1 Parent(s): f238cd4

Upload 2 files

Browse files
Files changed (2) hide show
  1. first_modality_generations_untrained.txt +793 -0
  2. model_architecture.txt +1977 -0
first_modality_generations_untrained.txt ADDED
@@ -0,0 +1,793 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Initializing ImageBind encoder ...
2
+ ... ImageBind encoder initialized.
3
+ Initializing Reasoner LLM model and tokenizer ...
4
+ Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
5
+ ... tokenizer initialized.
6
+ Freezing the Reasoner ...
7
+ ... Reasoner LLM model initialized.
8
+ Initializing input ImageBind Projection ...
9
+ ... Input ImageBind Projection initialized
10
+ text prompt: <|im_start|>system
11
+ You are a assistant<|im_end|>
12
+ <|im_start|>user
13
+ Describe the image and audio:
14
+
15
+ <|im_start|>system
16
+ You are a assistant<|im_end|>
17
+ <|im_start|>user
18
+ Describe the image and audio: <|image_start|> <|audio_start|>
19
+
20
+ recieved image
21
+ Encoding Image
22
+ [tensor([[[-2.0294e-03, 2.0996e-02, -1.3672e-02, -1.1475e-02, -1.9836e-03,
23
+ 9.7656e-03, -1.2451e-02, 8.0566e-03, -6.6757e-04, -7.4768e-04,
24
+ 1.3794e-02, 7.3853e-03, -2.2461e-02, 8.4305e-04, -2.1973e-02,
25
+ 8.3008e-03, 2.3193e-02, 1.3733e-02, -2.3926e-02, -2.3346e-03,
26
+ 3.5889e-02, 6.4697e-03, -1.2695e-02, 4.5471e-03, 2.0264e-02,
27
+ 9.9487e-03, 1.1841e-02, -1.5137e-02, 1.3672e-02, -1.9653e-02,
28
+ -6.3782e-03, 1.8921e-03, -3.6774e-03, 1.3672e-02, 5.3101e-03,
29
+ -2.3682e-02, 1.3245e-02, 1.2756e-02, 1.5182e-03, -4.6997e-03,
30
+ 1.5198e-02, 8.2397e-03, -1.0986e-02, 6.5918e-03, -4.6158e-04,
31
+ 1.0498e-02, 3.4027e-03, 1.6556e-03, 7.1716e-03, -3.3875e-03,
32
+ 2.9297e-03, 4.3945e-03, -3.4332e-03, 5.7068e-03, 3.8910e-03,
33
+ -1.7944e-02, -6.8054e-03, 2.4902e-02, -3.3691e-02, 2.8229e-04,
34
+ 5.7373e-03, -2.8687e-02, 5.0659e-03, 5.0049e-03, -2.2461e-02,
35
+ -1.5625e-02, -9.1553e-03, 1.0910e-03, -6.8359e-03, -4.3030e-03,
36
+ -2.2278e-03, 1.9043e-02, -1.6357e-02, 1.5411e-03, -3.5400e-02,
37
+ -7.2861e-04, 1.3977e-02, -5.3467e-02, -6.2561e-03, 2.3499e-03,
38
+ -1.3489e-02, 2.5177e-03, -5.7983e-03, -1.6357e-02, -4.4556e-03,
39
+ 1.3367e-02, 2.8442e-02, 2.0386e-02, -1.4465e-02, -1.5869e-02,
40
+ 2.1240e-02, -6.5308e-03, -9.8267e-03, -3.1250e-02, -1.3428e-03,
41
+ 1.1719e-02, 2.5024e-02, 2.1240e-02, 1.0681e-02, -3.9062e-02,
42
+ 4.3335e-03, 1.8555e-02, -9.3384e-03, -1.6357e-02, -1.6724e-02,
43
+ 2.1820e-03, -2.1362e-02, 3.4668e-02, -1.8555e-02, 3.5400e-03,
44
+ -4.9744e-03, -1.1536e-02, 4.3869e-04, -4.3335e-03, -5.7983e-03,
45
+ 2.7008e-03, 1.3351e-04, -5.4626e-03, 1.3000e-02, -1.1597e-02,
46
+ 2.1973e-03, 2.0020e-02, -8.7357e-04, 4.9744e-03, -4.4556e-03,
47
+ 7.7515e-03, -1.6602e-02, -5.3406e-03, -8.3008e-03, 1.1597e-02,
48
+ -2.5391e-02, -4.7607e-03, 1.4709e-02, 2.7222e-02, 1.8433e-02,
49
+ -1.8921e-02, -1.3916e-02, -8.6060e-03, -1.3916e-02, -1.9653e-02,
50
+ -6.9580e-03, -1.0498e-02, -1.3062e-02, 2.7466e-02, 3.4637e-03,
51
+ 8.9722e-03, 8.4229e-03, -1.4404e-02, -4.6692e-03, -4.1199e-03,
52
+ 8.4839e-03, -1.9531e-03, -5.2490e-03, -3.2043e-03, 4.7302e-03,
53
+ -5.4932e-03, 3.1738e-02, -1.2268e-02, 1.1597e-02, -1.6785e-03,
54
+ 2.7161e-03, -1.3199e-03, 1.6113e-02, -5.9814e-03, 1.0376e-02,
55
+ 2.7710e-02, 8.2397e-03, 1.8188e-02, -2.4170e-02, -5.0068e-05,
56
+ -4.6997e-03, 2.2339e-02, -1.0071e-03, -4.6082e-03, -6.6223e-03,
57
+ 1.1353e-02, 9.5367e-04, 2.5269e-02, -4.6387e-03, -3.2043e-04,
58
+ 1.3809e-03, -5.0659e-03, -2.2583e-02, -1.5015e-02, -1.0254e-02,
59
+ 2.6941e-05, 6.5002e-03, 1.1108e-02, -9.5215e-03, -1.5625e-02,
60
+ 1.8188e-02, 1.9169e-04, 7.5684e-03, 1.1353e-02, 2.6512e-04,
61
+ 5.3955e-02, 8.6670e-03, 3.5400e-03, 4.5776e-03, -8.5449e-03,
62
+ -7.5989e-03, -4.8828e-03, -2.2736e-03, -1.0010e-02, -1.0620e-02,
63
+ 2.7618e-03, -1.3184e-02, -1.9775e-02, 2.8198e-02, 3.6469e-03,
64
+ -2.2125e-03, 1.0803e-02, -4.2419e-03, 2.6489e-02, 1.6846e-02,
65
+ 6.4468e-04, -2.1362e-02, -1.1475e-02, 1.4038e-03, -1.4832e-02,
66
+ -1.2024e-02, 3.3203e-02, -1.9775e-02, 1.5831e-04, 1.4572e-03,
67
+ -1.4038e-02, 1.5564e-02, 3.3188e-04, 2.4292e-02, -6.4087e-03,
68
+ -1.5869e-02, 8.1177e-03, -1.1169e-02, 2.6489e-02, -6.7749e-03,
69
+ -7.5378e-03, 3.2227e-02, 2.7588e-02, 3.7079e-03, -1.0071e-02,
70
+ -2.3804e-03, 3.2959e-03, 4.7607e-03, -1.3367e-02, 4.6997e-03,
71
+ 1.0910e-03, -9.7046e-03, 2.0630e-02, -4.3335e-03, -8.9264e-04,
72
+ 6.2866e-03, -2.0264e-02, 2.7771e-03, 1.6724e-02, -1.4771e-02,
73
+ 6.6223e-03, 2.9297e-03, 1.1780e-02, -5.6076e-04, -1.6602e-02,
74
+ -4.9438e-03, -1.1719e-02, -2.7100e-02, -4.1016e-02, -8.7280e-03,
75
+ 1.2085e-02, 1.4465e-02, 1.5259e-03, 2.2949e-02, 7.5989e-03,
76
+ -2.0905e-03, 1.8433e-02, 5.9814e-03, 1.0010e-02, -1.8677e-02,
77
+ 1.2268e-02, 4.7302e-03, 4.3945e-03, 1.4526e-02, -7.4158e-03,
78
+ -2.7008e-03, 9.0942e-03, -3.8300e-03, -1.1414e-02, -1.1742e-05,
79
+ 1.6357e-02, 3.8910e-03, -1.7471e-03, 1.8845e-03, -1.0071e-02,
80
+ 4.4250e-03, 2.5391e-02, 1.1597e-02, 2.5146e-02, 6.8665e-03,
81
+ 1.9531e-02, -6.8359e-03, -1.2817e-02, -7.4768e-03, -1.5747e-02,
82
+ -2.3937e-04, 2.5635e-03, -1.9775e-02, -2.2461e-02, 2.0508e-02,
83
+ 9.1553e-03, 1.8158e-03, -2.0504e-04, -1.0864e-02, -1.5198e-02,
84
+ -1.3199e-03, 9.5825e-03, -3.0518e-03, 7.7515e-03, 2.0142e-02,
85
+ 1.7212e-02, -4.6158e-04, 9.1553e-03, -6.1951e-03, -5.5542e-03,
86
+ -2.2339e-02, -3.4332e-03, -1.6235e-02, 1.8677e-02, 1.8555e-02,
87
+ 8.7280e-03, 7.8735e-03, 1.1353e-02, -5.4321e-03, 2.0142e-02,
88
+ -8.8501e-03, -8.7280e-03, -1.4893e-02, -6.0120e-03, -1.2970e-04,
89
+ 2.1240e-02, 2.2705e-02, -3.3569e-04, 2.6367e-02, -5.6763e-03,
90
+ 1.2024e-02, -1.8555e-02, 2.1667e-03, -1.0376e-02, -5.4321e-03,
91
+ -1.2695e-02, -1.1719e-02, 1.4038e-02, -2.7084e-04, 4.4250e-03,
92
+ -6.6223e-03, -2.1515e-03, 1.4221e-02, 1.9531e-02, -3.0518e-04,
93
+ -1.9531e-02, 7.8735e-03, 1.6357e-02, 1.5991e-02, -7.2632e-03,
94
+ 2.6093e-03, 1.6479e-02, 8.0566e-03, -9.0942e-03, 8.4229e-03,
95
+ -1.9531e-02, -1.6602e-02, -2.4414e-02, -1.1963e-02, -1.9550e-04,
96
+ 6.7139e-03, -8.7280e-03, -8.6670e-03, -2.9053e-02, -4.8523e-03,
97
+ 7.5989e-03, -9.0942e-03, 5.5847e-03, -1.3794e-02, -1.4526e-02,
98
+ 9.3384e-03, -4.0283e-03, 5.1270e-03, -6.9427e-04, 9.3384e-03,
99
+ -1.3672e-02, 3.9101e-04, 1.1719e-02, -3.0823e-03, 1.1230e-02,
100
+ 3.9368e-03, 1.7212e-02, -1.3504e-03, 5.9509e-03, -9.1171e-04,
101
+ 2.2217e-02, -2.5635e-02, -1.2329e-02, -6.1646e-03, -5.3223e-02,
102
+ 1.4343e-03, 1.5076e-02, 2.7008e-03, -2.6611e-02, -1.0742e-02,
103
+ 1.5869e-02, 4.3640e-03, -5.7983e-03, 3.0884e-02, -5.0049e-03,
104
+ -1.0376e-02, -1.1292e-02, 2.4902e-02, 6.0730e-03, 4.5776e-03,
105
+ -1.8799e-02, -4.6997e-03, -5.5542e-03, 1.2390e-02, 8.7891e-03,
106
+ -7.8735e-03, -2.3193e-02, 7.8125e-03, -2.5513e-02, -1.8555e-02,
107
+ -2.0752e-03, -1.3611e-02, 3.1128e-02, -2.0996e-02, 9.8419e-04,
108
+ 3.1250e-02, -1.2634e-02, -1.6724e-02, 1.1414e-02, -1.3550e-02,
109
+ 1.7944e-02, -1.3733e-02, 2.4414e-02, 1.6968e-02, 1.6235e-02,
110
+ -2.3560e-02, -1.7578e-02, -2.3804e-03, -1.6479e-02, -1.6235e-02,
111
+ 1.8799e-02, 1.1353e-02, -1.0620e-02, 1.7090e-02, 2.7344e-02,
112
+ -1.2939e-02, 7.3853e-03, 1.7212e-02, 1.1536e-02, 1.6708e-03,
113
+ 8.7280e-03, 1.3245e-02, 9.8267e-03, -5.3406e-03, 1.8845e-03,
114
+ 9.5215e-03, -1.2634e-02, -4.0527e-02, 1.4160e-02, -1.1169e-02,
115
+ -2.0020e-02, -1.7822e-02, -7.5989e-03, -3.6621e-03, 1.4191e-03,
116
+ 3.0151e-02, -1.0315e-02, 2.2095e-02, -1.9165e-02, -3.1738e-02,
117
+ -1.3428e-02, 7.5684e-03, 3.3691e-02, 1.3428e-02, -1.1826e-03,
118
+ -3.7384e-03, -1.3672e-02, -4.4556e-03, -2.4536e-02, 8.7280e-03,
119
+ 7.2937e-03, 1.6212e-05, 1.0254e-02, 9.4604e-03, 1.2512e-02,
120
+ -7.0496e-03, -4.0588e-03, -4.4250e-03, -1.5869e-02, 2.8809e-02,
121
+ -2.5368e-04, 1.7090e-02, -9.2163e-03, 9.0332e-03, 1.1597e-02,
122
+ 4.4861e-03, 9.1553e-03, -5.8594e-03, -2.4414e-03, -1.0498e-02,
123
+ -1.6602e-02, -5.8594e-03, -4.6082e-03, -3.9673e-03, 1.1414e-02,
124
+ 3.4637e-03, 2.0020e-02, 1.7212e-02, -1.1414e-02, -2.9297e-03,
125
+ 1.8433e-02, 9.7656e-03, 4.9133e-03, 2.2705e-02, 2.4261e-03,
126
+ -6.1035e-03, 3.4027e-03, 1.7014e-03, -1.0742e-02, -6.5430e-02,
127
+ -1.3306e-02, -2.1484e-02, 1.6968e-02, -9.1553e-03, -5.5542e-03,
128
+ 9.2163e-03, -2.0142e-03, -1.8188e-02, -3.4904e-04, -3.2959e-02,
129
+ -1.9409e-02, 5.9204e-03, 1.6937e-03, -7.9346e-03, -8.9722e-03,
130
+ 2.5635e-03, -9.1553e-03, 1.8555e-02, -4.2725e-03, -3.2715e-02,
131
+ 1.7090e-02, 2.6855e-03, -3.3760e-04, 1.1902e-02, 8.4229e-03,
132
+ -4.4250e-04, -1.0803e-02, 2.2949e-02, 8.9111e-03, 8.4229e-03,
133
+ 1.8799e-02, 1.4954e-02, 5.3406e-03, 6.2561e-03, 6.0120e-03,
134
+ 5.2490e-03, -1.7944e-02, -1.5503e-02, -3.1281e-03, -1.2817e-02,
135
+ -1.9043e-02, 9.8419e-04, -4.3945e-03, 1.9897e-02, 6.0120e-03,
136
+ 5.4199e-02, 1.7853e-03, 6.8665e-04, -1.7334e-02, -2.3193e-02,
137
+ 6.7444e-03, 1.8188e-02, 4.2114e-03, -3.2227e-02, 1.1902e-02,
138
+ 1.5869e-02, -8.7280e-03, 7.5531e-04, 1.4709e-02, 1.0681e-02,
139
+ -5.9814e-03, -6.7444e-03, 1.1414e-02, 1.8555e-02, 9.7046e-03,
140
+ -2.2705e-02, -1.0376e-02, -1.7456e-02, -5.0659e-03, -6.4697e-03,
141
+ -2.0020e-02, 5.1270e-03, 1.7578e-02, -3.7689e-03, 1.5015e-02,
142
+ -4.1504e-03, 2.8320e-02, 1.0803e-02, 2.8229e-03, -2.0752e-02,
143
+ -1.7456e-02, 2.5787e-03, -2.2583e-02, -1.3733e-02, 2.9602e-03,
144
+ 1.6357e-02, -9.0332e-03, 4.6692e-03, -9.7656e-03, 4.6921e-04,
145
+ 5.7373e-03, 1.1475e-02, 3.8528e-04, 1.0803e-02, -6.3477e-03,
146
+ 4.3030e-03, -2.0630e-02, 1.1719e-02, 2.6245e-02, -1.9897e-02,
147
+ 9.8877e-03, -3.5706e-03, 8.7357e-04, -5.7068e-03, -1.1047e-02,
148
+ 2.2888e-03, 4.9438e-03, 2.2888e-03, 4.5471e-03, -2.0752e-03,
149
+ 6.9275e-03, -2.0142e-02, 3.2471e-02, 1.3306e-02, -7.8735e-03,
150
+ -1.7456e-02, 2.1118e-02, 2.6611e-02, 3.4912e-02, 1.0437e-02,
151
+ 2.8687e-03, 2.3438e-02, 2.9175e-02, 2.3560e-02, -4.5166e-03,
152
+ -2.1973e-02, 3.0518e-02, -3.9673e-03, 6.4850e-04, 7.1716e-03,
153
+ -7.8735e-03, -2.6245e-03, 1.9531e-02, -2.4414e-03, 1.6113e-02,
154
+ 3.0029e-02, 4.6997e-03, 5.7983e-03, 5.6763e-03, 3.7231e-03,
155
+ 1.4648e-02, -1.2817e-02, -6.9275e-03, 3.5858e-03, -2.0386e-02,
156
+ 1.9775e-02, 1.3611e-02, 1.6113e-02, -9.2773e-03, -1.5442e-02,
157
+ 1.9775e-02, -2.8381e-03, 1.2695e-02, 1.3306e-02, 2.9449e-03,
158
+ 1.0925e-02, -1.2207e-02, -1.1475e-02, -1.1520e-03, -1.2970e-03,
159
+ 8.3618e-03, 1.3046e-03, -1.3885e-03, -5.0354e-03, -1.0986e-02,
160
+ -2.2827e-02, -1.3367e-02, -3.0029e-02, 1.3657e-03, 1.0376e-02,
161
+ -1.8188e-02, -8.4839e-03, 7.5989e-03, 1.4832e-02, 2.5269e-02,
162
+ -6.0272e-04, 1.4954e-02, 3.9673e-03, 5.0049e-03, 3.0151e-02,
163
+ -7.2327e-03, -1.7944e-02, 2.2217e-02, 5.6152e-03, -1.1215e-03,
164
+ -8.3618e-03, -2.2461e-02, 1.1597e-02, -2.3499e-03, 7.8125e-03,
165
+ 1.5381e-02, -1.1292e-03, 3.0670e-03, 1.9989e-03, 9.5825e-03,
166
+ 5.7068e-03, -8.1787e-03, -2.6978e-02, 8.1787e-03, 1.4420e-03,
167
+ -8.4229e-03, -9.9182e-04, -8.3618e-03, -2.4261e-03, 6.4697e-03,
168
+ 1.5625e-02, 5.7983e-03, -1.6235e-02, 5.0049e-03, 2.7954e-02,
169
+ 1.8677e-02, 3.0762e-02, 1.6785e-03, -1.8555e-02, -1.2146e-02,
170
+ -1.5503e-02, 4.4250e-03, 1.4526e-02, -1.1536e-02, -1.0864e-02,
171
+ 2.0630e-02, -2.4658e-02, -1.5442e-02, -1.2207e-02, -2.3926e-02,
172
+ 8.5449e-03, -1.6846e-02, -5.2795e-03, 1.1353e-02, 5.1270e-03,
173
+ 1.4496e-03, -1.1719e-02, -2.4567e-03, 6.4969e-06, -1.9684e-03,
174
+ -2.3438e-02, 8.3008e-03, -7.9346e-03, -1.5747e-02, 2.5879e-02,
175
+ 2.1210e-03, -5.8289e-03, -4.3945e-03, -5.1270e-03, -3.3569e-03,
176
+ 1.2146e-02, -2.7466e-02, 1.7700e-02, 7.3853e-03, 1.5869e-02,
177
+ -1.1292e-02, 2.4414e-02, -7.1411e-03, -6.7139e-03, -1.5974e-05,
178
+ -1.0681e-02, 1.2207e-02, 5.1575e-03, 1.6479e-03, -1.9989e-03,
179
+ 4.5471e-03, 4.6082e-03, 1.1169e-02, 1.3733e-03, -1.3504e-03,
180
+ -8.4686e-04, 2.6245e-02, -2.0752e-03, 1.1368e-03, -5.3406e-03,
181
+ 7.9956e-03, -1.6357e-02, 6.9580e-03, 1.2756e-02, -2.7344e-02,
182
+ -6.5613e-03, 2.8839e-03, -1.3367e-02, -1.7212e-02, -1.4526e-02,
183
+ -9.2163e-03, 2.0630e-02, 5.4016e-03, 2.2949e-02, 7.9956e-03,
184
+ -3.4180e-03, -2.4780e-02, -3.2654e-03, 1.0147e-03, 1.4221e-02,
185
+ 6.2256e-03, -1.7242e-03, -1.9531e-02, -3.4943e-03, -9.3384e-03,
186
+ -1.5869e-02, 1.1902e-03, -1.1902e-02, -7.6599e-03, -1.8799e-02,
187
+ 3.6774e-03, -6.8970e-03, 1.2939e-02, -4.3945e-03, 3.3203e-02,
188
+ -2.4048e-02, 3.0212e-03, -4.8523e-03, 3.4027e-03, -9.1934e-04,
189
+ 1.3306e-02, 3.2959e-02, -2.1606e-02, -1.8799e-02, -9.6436e-03,
190
+ -1.1047e-02, -8.6060e-03, 1.7334e-02, -1.0193e-02, 8.2397e-03,
191
+ 4.0894e-03, -7.6599e-03, -4.6158e-04, -6.1951e-03, 8.9722e-03,
192
+ 1.7090e-03, 2.5513e-02, 1.1719e-02, -1.1719e-02, 9.3384e-03,
193
+ -7.5989e-03, 1.7578e-02, 5.2795e-03, 1.3306e-02, -1.0986e-02,
194
+ 3.1281e-03, -1.8066e-02, -1.5320e-02, 1.7090e-03, 1.2451e-02,
195
+ 3.2654e-03, 4.2236e-02, 1.4221e-02, -1.5381e-02, 1.6724e-02,
196
+ -2.4536e-02, -2.3651e-03, 3.0060e-03, 8.0566e-03, 1.1963e-02,
197
+ 2.8076e-03, -7.5684e-03, 1.0254e-02, 2.1851e-02, -1.2756e-02,
198
+ 1.5625e-02, 2.8564e-02, -2.9144e-03, -5.4016e-03, 2.1362e-02,
199
+ 9.8877e-03, 1.4954e-02, 2.6855e-02, -1.0559e-02, -1.1536e-02,
200
+ 8.1787e-03, -9.9487e-03, 7.3853e-03, 5.7068e-03, -1.8433e-02,
201
+ 1.1963e-02]]]), tensor([[[-0.0020, 0.0210, -0.0137, ..., 0.0057, -0.0184, 0.0120],
202
+ [-0.0120, -0.0040, 0.0083, ..., 0.0083, 0.0131, -0.0119],
203
+ [ 0.0035, -0.0315, -0.0108, ..., 0.0160, 0.0194, -0.0033],
204
+ ...,
205
+ [ 0.0134, 0.0093, 0.0005, ..., -0.0001, 0.0204, 0.0117],
206
+ [ 0.0061, -0.0312, 0.0286, ..., 0.0228, 0.0232, -0.0014],
207
+ [-0.0030, -0.0425, 0.0232, ..., 0.0081, 0.0537, -0.0090]]]), tensor([[[-0.0250, -0.0245, 0.0175, ..., 0.0332, 0.0106, 0.0041],
208
+ [ 0.0060, -0.0280, 0.0079, ..., -0.0086, 0.0168, 0.0135],
209
+ [-0.0121, 0.0053, 0.0112, ..., 0.0165, -0.0311, 0.0092],
210
+ ...,
211
+ [ 0.0245, 0.0073, 0.0044, ..., 0.0104, -0.0150, -0.0141],
212
+ [ 0.0060, -0.0280, 0.0079, ..., -0.0086, 0.0168, 0.0135],
213
+ [ 0.0123, -0.0060, 0.0310, ..., 0.0021, 0.0104, 0.0239]]],
214
+ grad_fn=<CatBackward0>)]
215
+
216
+ recieved audio
217
+ Encoding Audio
218
+ [tensor([[[-2.0294e-03, 2.0996e-02, -1.3672e-02, -1.1475e-02, -1.9836e-03,
219
+ 9.7656e-03, -1.2451e-02, 8.0566e-03, -6.6757e-04, -7.4768e-04,
220
+ 1.3794e-02, 7.3853e-03, -2.2461e-02, 8.4305e-04, -2.1973e-02,
221
+ 8.3008e-03, 2.3193e-02, 1.3733e-02, -2.3926e-02, -2.3346e-03,
222
+ 3.5889e-02, 6.4697e-03, -1.2695e-02, 4.5471e-03, 2.0264e-02,
223
+ 9.9487e-03, 1.1841e-02, -1.5137e-02, 1.3672e-02, -1.9653e-02,
224
+ -6.3782e-03, 1.8921e-03, -3.6774e-03, 1.3672e-02, 5.3101e-03,
225
+ -2.3682e-02, 1.3245e-02, 1.2756e-02, 1.5182e-03, -4.6997e-03,
226
+ 1.5198e-02, 8.2397e-03, -1.0986e-02, 6.5918e-03, -4.6158e-04,
227
+ 1.0498e-02, 3.4027e-03, 1.6556e-03, 7.1716e-03, -3.3875e-03,
228
+ 2.9297e-03, 4.3945e-03, -3.4332e-03, 5.7068e-03, 3.8910e-03,
229
+ -1.7944e-02, -6.8054e-03, 2.4902e-02, -3.3691e-02, 2.8229e-04,
230
+ 5.7373e-03, -2.8687e-02, 5.0659e-03, 5.0049e-03, -2.2461e-02,
231
+ -1.5625e-02, -9.1553e-03, 1.0910e-03, -6.8359e-03, -4.3030e-03,
232
+ -2.2278e-03, 1.9043e-02, -1.6357e-02, 1.5411e-03, -3.5400e-02,
233
+ -7.2861e-04, 1.3977e-02, -5.3467e-02, -6.2561e-03, 2.3499e-03,
234
+ -1.3489e-02, 2.5177e-03, -5.7983e-03, -1.6357e-02, -4.4556e-03,
235
+ 1.3367e-02, 2.8442e-02, 2.0386e-02, -1.4465e-02, -1.5869e-02,
236
+ 2.1240e-02, -6.5308e-03, -9.8267e-03, -3.1250e-02, -1.3428e-03,
237
+ 1.1719e-02, 2.5024e-02, 2.1240e-02, 1.0681e-02, -3.9062e-02,
238
+ 4.3335e-03, 1.8555e-02, -9.3384e-03, -1.6357e-02, -1.6724e-02,
239
+ 2.1820e-03, -2.1362e-02, 3.4668e-02, -1.8555e-02, 3.5400e-03,
240
+ -4.9744e-03, -1.1536e-02, 4.3869e-04, -4.3335e-03, -5.7983e-03,
241
+ 2.7008e-03, 1.3351e-04, -5.4626e-03, 1.3000e-02, -1.1597e-02,
242
+ 2.1973e-03, 2.0020e-02, -8.7357e-04, 4.9744e-03, -4.4556e-03,
243
+ 7.7515e-03, -1.6602e-02, -5.3406e-03, -8.3008e-03, 1.1597e-02,
244
+ -2.5391e-02, -4.7607e-03, 1.4709e-02, 2.7222e-02, 1.8433e-02,
245
+ -1.8921e-02, -1.3916e-02, -8.6060e-03, -1.3916e-02, -1.9653e-02,
246
+ -6.9580e-03, -1.0498e-02, -1.3062e-02, 2.7466e-02, 3.4637e-03,
247
+ 8.9722e-03, 8.4229e-03, -1.4404e-02, -4.6692e-03, -4.1199e-03,
248
+ 8.4839e-03, -1.9531e-03, -5.2490e-03, -3.2043e-03, 4.7302e-03,
249
+ -5.4932e-03, 3.1738e-02, -1.2268e-02, 1.1597e-02, -1.6785e-03,
250
+ 2.7161e-03, -1.3199e-03, 1.6113e-02, -5.9814e-03, 1.0376e-02,
251
+ 2.7710e-02, 8.2397e-03, 1.8188e-02, -2.4170e-02, -5.0068e-05,
252
+ -4.6997e-03, 2.2339e-02, -1.0071e-03, -4.6082e-03, -6.6223e-03,
253
+ 1.1353e-02, 9.5367e-04, 2.5269e-02, -4.6387e-03, -3.2043e-04,
254
+ 1.3809e-03, -5.0659e-03, -2.2583e-02, -1.5015e-02, -1.0254e-02,
255
+ 2.6941e-05, 6.5002e-03, 1.1108e-02, -9.5215e-03, -1.5625e-02,
256
+ 1.8188e-02, 1.9169e-04, 7.5684e-03, 1.1353e-02, 2.6512e-04,
257
+ 5.3955e-02, 8.6670e-03, 3.5400e-03, 4.5776e-03, -8.5449e-03,
258
+ -7.5989e-03, -4.8828e-03, -2.2736e-03, -1.0010e-02, -1.0620e-02,
259
+ 2.7618e-03, -1.3184e-02, -1.9775e-02, 2.8198e-02, 3.6469e-03,
260
+ -2.2125e-03, 1.0803e-02, -4.2419e-03, 2.6489e-02, 1.6846e-02,
261
+ 6.4468e-04, -2.1362e-02, -1.1475e-02, 1.4038e-03, -1.4832e-02,
262
+ -1.2024e-02, 3.3203e-02, -1.9775e-02, 1.5831e-04, 1.4572e-03,
263
+ -1.4038e-02, 1.5564e-02, 3.3188e-04, 2.4292e-02, -6.4087e-03,
264
+ -1.5869e-02, 8.1177e-03, -1.1169e-02, 2.6489e-02, -6.7749e-03,
265
+ -7.5378e-03, 3.2227e-02, 2.7588e-02, 3.7079e-03, -1.0071e-02,
266
+ -2.3804e-03, 3.2959e-03, 4.7607e-03, -1.3367e-02, 4.6997e-03,
267
+ 1.0910e-03, -9.7046e-03, 2.0630e-02, -4.3335e-03, -8.9264e-04,
268
+ 6.2866e-03, -2.0264e-02, 2.7771e-03, 1.6724e-02, -1.4771e-02,
269
+ 6.6223e-03, 2.9297e-03, 1.1780e-02, -5.6076e-04, -1.6602e-02,
270
+ -4.9438e-03, -1.1719e-02, -2.7100e-02, -4.1016e-02, -8.7280e-03,
271
+ 1.2085e-02, 1.4465e-02, 1.5259e-03, 2.2949e-02, 7.5989e-03,
272
+ -2.0905e-03, 1.8433e-02, 5.9814e-03, 1.0010e-02, -1.8677e-02,
273
+ 1.2268e-02, 4.7302e-03, 4.3945e-03, 1.4526e-02, -7.4158e-03,
274
+ -2.7008e-03, 9.0942e-03, -3.8300e-03, -1.1414e-02, -1.1742e-05,
275
+ 1.6357e-02, 3.8910e-03, -1.7471e-03, 1.8845e-03, -1.0071e-02,
276
+ 4.4250e-03, 2.5391e-02, 1.1597e-02, 2.5146e-02, 6.8665e-03,
277
+ 1.9531e-02, -6.8359e-03, -1.2817e-02, -7.4768e-03, -1.5747e-02,
278
+ -2.3937e-04, 2.5635e-03, -1.9775e-02, -2.2461e-02, 2.0508e-02,
279
+ 9.1553e-03, 1.8158e-03, -2.0504e-04, -1.0864e-02, -1.5198e-02,
280
+ -1.3199e-03, 9.5825e-03, -3.0518e-03, 7.7515e-03, 2.0142e-02,
281
+ 1.7212e-02, -4.6158e-04, 9.1553e-03, -6.1951e-03, -5.5542e-03,
282
+ -2.2339e-02, -3.4332e-03, -1.6235e-02, 1.8677e-02, 1.8555e-02,
283
+ 8.7280e-03, 7.8735e-03, 1.1353e-02, -5.4321e-03, 2.0142e-02,
284
+ -8.8501e-03, -8.7280e-03, -1.4893e-02, -6.0120e-03, -1.2970e-04,
285
+ 2.1240e-02, 2.2705e-02, -3.3569e-04, 2.6367e-02, -5.6763e-03,
286
+ 1.2024e-02, -1.8555e-02, 2.1667e-03, -1.0376e-02, -5.4321e-03,
287
+ -1.2695e-02, -1.1719e-02, 1.4038e-02, -2.7084e-04, 4.4250e-03,
288
+ -6.6223e-03, -2.1515e-03, 1.4221e-02, 1.9531e-02, -3.0518e-04,
289
+ -1.9531e-02, 7.8735e-03, 1.6357e-02, 1.5991e-02, -7.2632e-03,
290
+ 2.6093e-03, 1.6479e-02, 8.0566e-03, -9.0942e-03, 8.4229e-03,
291
+ -1.9531e-02, -1.6602e-02, -2.4414e-02, -1.1963e-02, -1.9550e-04,
292
+ 6.7139e-03, -8.7280e-03, -8.6670e-03, -2.9053e-02, -4.8523e-03,
293
+ 7.5989e-03, -9.0942e-03, 5.5847e-03, -1.3794e-02, -1.4526e-02,
294
+ 9.3384e-03, -4.0283e-03, 5.1270e-03, -6.9427e-04, 9.3384e-03,
295
+ -1.3672e-02, 3.9101e-04, 1.1719e-02, -3.0823e-03, 1.1230e-02,
296
+ 3.9368e-03, 1.7212e-02, -1.3504e-03, 5.9509e-03, -9.1171e-04,
297
+ 2.2217e-02, -2.5635e-02, -1.2329e-02, -6.1646e-03, -5.3223e-02,
298
+ 1.4343e-03, 1.5076e-02, 2.7008e-03, -2.6611e-02, -1.0742e-02,
299
+ 1.5869e-02, 4.3640e-03, -5.7983e-03, 3.0884e-02, -5.0049e-03,
300
+ -1.0376e-02, -1.1292e-02, 2.4902e-02, 6.0730e-03, 4.5776e-03,
301
+ -1.8799e-02, -4.6997e-03, -5.5542e-03, 1.2390e-02, 8.7891e-03,
302
+ -7.8735e-03, -2.3193e-02, 7.8125e-03, -2.5513e-02, -1.8555e-02,
303
+ -2.0752e-03, -1.3611e-02, 3.1128e-02, -2.0996e-02, 9.8419e-04,
304
+ 3.1250e-02, -1.2634e-02, -1.6724e-02, 1.1414e-02, -1.3550e-02,
305
+ 1.7944e-02, -1.3733e-02, 2.4414e-02, 1.6968e-02, 1.6235e-02,
306
+ -2.3560e-02, -1.7578e-02, -2.3804e-03, -1.6479e-02, -1.6235e-02,
307
+ 1.8799e-02, 1.1353e-02, -1.0620e-02, 1.7090e-02, 2.7344e-02,
308
+ -1.2939e-02, 7.3853e-03, 1.7212e-02, 1.1536e-02, 1.6708e-03,
309
+ 8.7280e-03, 1.3245e-02, 9.8267e-03, -5.3406e-03, 1.8845e-03,
310
+ 9.5215e-03, -1.2634e-02, -4.0527e-02, 1.4160e-02, -1.1169e-02,
311
+ -2.0020e-02, -1.7822e-02, -7.5989e-03, -3.6621e-03, 1.4191e-03,
312
+ 3.0151e-02, -1.0315e-02, 2.2095e-02, -1.9165e-02, -3.1738e-02,
313
+ -1.3428e-02, 7.5684e-03, 3.3691e-02, 1.3428e-02, -1.1826e-03,
314
+ -3.7384e-03, -1.3672e-02, -4.4556e-03, -2.4536e-02, 8.7280e-03,
315
+ 7.2937e-03, 1.6212e-05, 1.0254e-02, 9.4604e-03, 1.2512e-02,
316
+ -7.0496e-03, -4.0588e-03, -4.4250e-03, -1.5869e-02, 2.8809e-02,
317
+ -2.5368e-04, 1.7090e-02, -9.2163e-03, 9.0332e-03, 1.1597e-02,
318
+ 4.4861e-03, 9.1553e-03, -5.8594e-03, -2.4414e-03, -1.0498e-02,
319
+ -1.6602e-02, -5.8594e-03, -4.6082e-03, -3.9673e-03, 1.1414e-02,
320
+ 3.4637e-03, 2.0020e-02, 1.7212e-02, -1.1414e-02, -2.9297e-03,
321
+ 1.8433e-02, 9.7656e-03, 4.9133e-03, 2.2705e-02, 2.4261e-03,
322
+ -6.1035e-03, 3.4027e-03, 1.7014e-03, -1.0742e-02, -6.5430e-02,
323
+ -1.3306e-02, -2.1484e-02, 1.6968e-02, -9.1553e-03, -5.5542e-03,
324
+ 9.2163e-03, -2.0142e-03, -1.8188e-02, -3.4904e-04, -3.2959e-02,
325
+ -1.9409e-02, 5.9204e-03, 1.6937e-03, -7.9346e-03, -8.9722e-03,
326
+ 2.5635e-03, -9.1553e-03, 1.8555e-02, -4.2725e-03, -3.2715e-02,
327
+ 1.7090e-02, 2.6855e-03, -3.3760e-04, 1.1902e-02, 8.4229e-03,
328
+ -4.4250e-04, -1.0803e-02, 2.2949e-02, 8.9111e-03, 8.4229e-03,
329
+ 1.8799e-02, 1.4954e-02, 5.3406e-03, 6.2561e-03, 6.0120e-03,
330
+ 5.2490e-03, -1.7944e-02, -1.5503e-02, -3.1281e-03, -1.2817e-02,
331
+ -1.9043e-02, 9.8419e-04, -4.3945e-03, 1.9897e-02, 6.0120e-03,
332
+ 5.4199e-02, 1.7853e-03, 6.8665e-04, -1.7334e-02, -2.3193e-02,
333
+ 6.7444e-03, 1.8188e-02, 4.2114e-03, -3.2227e-02, 1.1902e-02,
334
+ 1.5869e-02, -8.7280e-03, 7.5531e-04, 1.4709e-02, 1.0681e-02,
335
+ -5.9814e-03, -6.7444e-03, 1.1414e-02, 1.8555e-02, 9.7046e-03,
336
+ -2.2705e-02, -1.0376e-02, -1.7456e-02, -5.0659e-03, -6.4697e-03,
337
+ -2.0020e-02, 5.1270e-03, 1.7578e-02, -3.7689e-03, 1.5015e-02,
338
+ -4.1504e-03, 2.8320e-02, 1.0803e-02, 2.8229e-03, -2.0752e-02,
339
+ -1.7456e-02, 2.5787e-03, -2.2583e-02, -1.3733e-02, 2.9602e-03,
340
+ 1.6357e-02, -9.0332e-03, 4.6692e-03, -9.7656e-03, 4.6921e-04,
341
+ 5.7373e-03, 1.1475e-02, 3.8528e-04, 1.0803e-02, -6.3477e-03,
342
+ 4.3030e-03, -2.0630e-02, 1.1719e-02, 2.6245e-02, -1.9897e-02,
343
+ 9.8877e-03, -3.5706e-03, 8.7357e-04, -5.7068e-03, -1.1047e-02,
344
+ 2.2888e-03, 4.9438e-03, 2.2888e-03, 4.5471e-03, -2.0752e-03,
345
+ 6.9275e-03, -2.0142e-02, 3.2471e-02, 1.3306e-02, -7.8735e-03,
346
+ -1.7456e-02, 2.1118e-02, 2.6611e-02, 3.4912e-02, 1.0437e-02,
347
+ 2.8687e-03, 2.3438e-02, 2.9175e-02, 2.3560e-02, -4.5166e-03,
348
+ -2.1973e-02, 3.0518e-02, -3.9673e-03, 6.4850e-04, 7.1716e-03,
349
+ -7.8735e-03, -2.6245e-03, 1.9531e-02, -2.4414e-03, 1.6113e-02,
350
+ 3.0029e-02, 4.6997e-03, 5.7983e-03, 5.6763e-03, 3.7231e-03,
351
+ 1.4648e-02, -1.2817e-02, -6.9275e-03, 3.5858e-03, -2.0386e-02,
352
+ 1.9775e-02, 1.3611e-02, 1.6113e-02, -9.2773e-03, -1.5442e-02,
353
+ 1.9775e-02, -2.8381e-03, 1.2695e-02, 1.3306e-02, 2.9449e-03,
354
+ 1.0925e-02, -1.2207e-02, -1.1475e-02, -1.1520e-03, -1.2970e-03,
355
+ 8.3618e-03, 1.3046e-03, -1.3885e-03, -5.0354e-03, -1.0986e-02,
356
+ -2.2827e-02, -1.3367e-02, -3.0029e-02, 1.3657e-03, 1.0376e-02,
357
+ -1.8188e-02, -8.4839e-03, 7.5989e-03, 1.4832e-02, 2.5269e-02,
358
+ -6.0272e-04, 1.4954e-02, 3.9673e-03, 5.0049e-03, 3.0151e-02,
359
+ -7.2327e-03, -1.7944e-02, 2.2217e-02, 5.6152e-03, -1.1215e-03,
360
+ -8.3618e-03, -2.2461e-02, 1.1597e-02, -2.3499e-03, 7.8125e-03,
361
+ 1.5381e-02, -1.1292e-03, 3.0670e-03, 1.9989e-03, 9.5825e-03,
362
+ 5.7068e-03, -8.1787e-03, -2.6978e-02, 8.1787e-03, 1.4420e-03,
363
+ -8.4229e-03, -9.9182e-04, -8.3618e-03, -2.4261e-03, 6.4697e-03,
364
+ 1.5625e-02, 5.7983e-03, -1.6235e-02, 5.0049e-03, 2.7954e-02,
365
+ 1.8677e-02, 3.0762e-02, 1.6785e-03, -1.8555e-02, -1.2146e-02,
366
+ -1.5503e-02, 4.4250e-03, 1.4526e-02, -1.1536e-02, -1.0864e-02,
367
+ 2.0630e-02, -2.4658e-02, -1.5442e-02, -1.2207e-02, -2.3926e-02,
368
+ 8.5449e-03, -1.6846e-02, -5.2795e-03, 1.1353e-02, 5.1270e-03,
369
+ 1.4496e-03, -1.1719e-02, -2.4567e-03, 6.4969e-06, -1.9684e-03,
370
+ -2.3438e-02, 8.3008e-03, -7.9346e-03, -1.5747e-02, 2.5879e-02,
371
+ 2.1210e-03, -5.8289e-03, -4.3945e-03, -5.1270e-03, -3.3569e-03,
372
+ 1.2146e-02, -2.7466e-02, 1.7700e-02, 7.3853e-03, 1.5869e-02,
373
+ -1.1292e-02, 2.4414e-02, -7.1411e-03, -6.7139e-03, -1.5974e-05,
374
+ -1.0681e-02, 1.2207e-02, 5.1575e-03, 1.6479e-03, -1.9989e-03,
375
+ 4.5471e-03, 4.6082e-03, 1.1169e-02, 1.3733e-03, -1.3504e-03,
376
+ -8.4686e-04, 2.6245e-02, -2.0752e-03, 1.1368e-03, -5.3406e-03,
377
+ 7.9956e-03, -1.6357e-02, 6.9580e-03, 1.2756e-02, -2.7344e-02,
378
+ -6.5613e-03, 2.8839e-03, -1.3367e-02, -1.7212e-02, -1.4526e-02,
379
+ -9.2163e-03, 2.0630e-02, 5.4016e-03, 2.2949e-02, 7.9956e-03,
380
+ -3.4180e-03, -2.4780e-02, -3.2654e-03, 1.0147e-03, 1.4221e-02,
381
+ 6.2256e-03, -1.7242e-03, -1.9531e-02, -3.4943e-03, -9.3384e-03,
382
+ -1.5869e-02, 1.1902e-03, -1.1902e-02, -7.6599e-03, -1.8799e-02,
383
+ 3.6774e-03, -6.8970e-03, 1.2939e-02, -4.3945e-03, 3.3203e-02,
384
+ -2.4048e-02, 3.0212e-03, -4.8523e-03, 3.4027e-03, -9.1934e-04,
385
+ 1.3306e-02, 3.2959e-02, -2.1606e-02, -1.8799e-02, -9.6436e-03,
386
+ -1.1047e-02, -8.6060e-03, 1.7334e-02, -1.0193e-02, 8.2397e-03,
387
+ 4.0894e-03, -7.6599e-03, -4.6158e-04, -6.1951e-03, 8.9722e-03,
388
+ 1.7090e-03, 2.5513e-02, 1.1719e-02, -1.1719e-02, 9.3384e-03,
389
+ -7.5989e-03, 1.7578e-02, 5.2795e-03, 1.3306e-02, -1.0986e-02,
390
+ 3.1281e-03, -1.8066e-02, -1.5320e-02, 1.7090e-03, 1.2451e-02,
391
+ 3.2654e-03, 4.2236e-02, 1.4221e-02, -1.5381e-02, 1.6724e-02,
392
+ -2.4536e-02, -2.3651e-03, 3.0060e-03, 8.0566e-03, 1.1963e-02,
393
+ 2.8076e-03, -7.5684e-03, 1.0254e-02, 2.1851e-02, -1.2756e-02,
394
+ 1.5625e-02, 2.8564e-02, -2.9144e-03, -5.4016e-03, 2.1362e-02,
395
+ 9.8877e-03, 1.4954e-02, 2.6855e-02, -1.0559e-02, -1.1536e-02,
396
+ 8.1787e-03, -9.9487e-03, 7.3853e-03, 5.7068e-03, -1.8433e-02,
397
+ 1.1963e-02]]]), tensor([[[-0.0020, 0.0210, -0.0137, ..., 0.0057, -0.0184, 0.0120],
398
+ [-0.0120, -0.0040, 0.0083, ..., 0.0083, 0.0131, -0.0119],
399
+ [ 0.0035, -0.0315, -0.0108, ..., 0.0160, 0.0194, -0.0033],
400
+ ...,
401
+ [ 0.0134, 0.0093, 0.0005, ..., -0.0001, 0.0204, 0.0117],
402
+ [ 0.0061, -0.0312, 0.0286, ..., 0.0228, 0.0232, -0.0014],
403
+ [-0.0030, -0.0425, 0.0232, ..., 0.0081, 0.0537, -0.0090]]]), tensor([[[-0.0250, -0.0245, 0.0175, ..., 0.0332, 0.0106, 0.0041],
404
+ [ 0.0060, -0.0280, 0.0079, ..., -0.0086, 0.0168, 0.0135],
405
+ [-0.0121, 0.0053, 0.0112, ..., 0.0165, -0.0311, 0.0092],
406
+ ...,
407
+ [ 0.0245, 0.0073, 0.0044, ..., 0.0104, -0.0150, -0.0141],
408
+ [ 0.0060, -0.0280, 0.0079, ..., -0.0086, 0.0168, 0.0135],
409
+ [ 0.0123, -0.0060, 0.0310, ..., 0.0021, 0.0104, 0.0239]]],
410
+ grad_fn=<CatBackward0>), tensor([[[-2.0294e-03, 2.0996e-02, -1.3672e-02, -1.1475e-02, -1.9836e-03,
411
+ 9.7656e-03, -1.2451e-02, 8.0566e-03, -6.6757e-04, -7.4768e-04,
412
+ 1.3794e-02, 7.3853e-03, -2.2461e-02, 8.4305e-04, -2.1973e-02,
413
+ 8.3008e-03, 2.3193e-02, 1.3733e-02, -2.3926e-02, -2.3346e-03,
414
+ 3.5889e-02, 6.4697e-03, -1.2695e-02, 4.5471e-03, 2.0264e-02,
415
+ 9.9487e-03, 1.1841e-02, -1.5137e-02, 1.3672e-02, -1.9653e-02,
416
+ -6.3782e-03, 1.8921e-03, -3.6774e-03, 1.3672e-02, 5.3101e-03,
417
+ -2.3682e-02, 1.3245e-02, 1.2756e-02, 1.5182e-03, -4.6997e-03,
418
+ 1.5198e-02, 8.2397e-03, -1.0986e-02, 6.5918e-03, -4.6158e-04,
419
+ 1.0498e-02, 3.4027e-03, 1.6556e-03, 7.1716e-03, -3.3875e-03,
420
+ 2.9297e-03, 4.3945e-03, -3.4332e-03, 5.7068e-03, 3.8910e-03,
421
+ -1.7944e-02, -6.8054e-03, 2.4902e-02, -3.3691e-02, 2.8229e-04,
422
+ 5.7373e-03, -2.8687e-02, 5.0659e-03, 5.0049e-03, -2.2461e-02,
423
+ -1.5625e-02, -9.1553e-03, 1.0910e-03, -6.8359e-03, -4.3030e-03,
424
+ -2.2278e-03, 1.9043e-02, -1.6357e-02, 1.5411e-03, -3.5400e-02,
425
+ -7.2861e-04, 1.3977e-02, -5.3467e-02, -6.2561e-03, 2.3499e-03,
426
+ -1.3489e-02, 2.5177e-03, -5.7983e-03, -1.6357e-02, -4.4556e-03,
427
+ 1.3367e-02, 2.8442e-02, 2.0386e-02, -1.4465e-02, -1.5869e-02,
428
+ 2.1240e-02, -6.5308e-03, -9.8267e-03, -3.1250e-02, -1.3428e-03,
429
+ 1.1719e-02, 2.5024e-02, 2.1240e-02, 1.0681e-02, -3.9062e-02,
430
+ 4.3335e-03, 1.8555e-02, -9.3384e-03, -1.6357e-02, -1.6724e-02,
431
+ 2.1820e-03, -2.1362e-02, 3.4668e-02, -1.8555e-02, 3.5400e-03,
432
+ -4.9744e-03, -1.1536e-02, 4.3869e-04, -4.3335e-03, -5.7983e-03,
433
+ 2.7008e-03, 1.3351e-04, -5.4626e-03, 1.3000e-02, -1.1597e-02,
434
+ 2.1973e-03, 2.0020e-02, -8.7357e-04, 4.9744e-03, -4.4556e-03,
435
+ 7.7515e-03, -1.6602e-02, -5.3406e-03, -8.3008e-03, 1.1597e-02,
436
+ -2.5391e-02, -4.7607e-03, 1.4709e-02, 2.7222e-02, 1.8433e-02,
437
+ -1.8921e-02, -1.3916e-02, -8.6060e-03, -1.3916e-02, -1.9653e-02,
438
+ -6.9580e-03, -1.0498e-02, -1.3062e-02, 2.7466e-02, 3.4637e-03,
439
+ 8.9722e-03, 8.4229e-03, -1.4404e-02, -4.6692e-03, -4.1199e-03,
440
+ 8.4839e-03, -1.9531e-03, -5.2490e-03, -3.2043e-03, 4.7302e-03,
441
+ -5.4932e-03, 3.1738e-02, -1.2268e-02, 1.1597e-02, -1.6785e-03,
442
+ 2.7161e-03, -1.3199e-03, 1.6113e-02, -5.9814e-03, 1.0376e-02,
443
+ 2.7710e-02, 8.2397e-03, 1.8188e-02, -2.4170e-02, -5.0068e-05,
444
+ -4.6997e-03, 2.2339e-02, -1.0071e-03, -4.6082e-03, -6.6223e-03,
445
+ 1.1353e-02, 9.5367e-04, 2.5269e-02, -4.6387e-03, -3.2043e-04,
446
+ 1.3809e-03, -5.0659e-03, -2.2583e-02, -1.5015e-02, -1.0254e-02,
447
+ 2.6941e-05, 6.5002e-03, 1.1108e-02, -9.5215e-03, -1.5625e-02,
448
+ 1.8188e-02, 1.9169e-04, 7.5684e-03, 1.1353e-02, 2.6512e-04,
449
+ 5.3955e-02, 8.6670e-03, 3.5400e-03, 4.5776e-03, -8.5449e-03,
450
+ -7.5989e-03, -4.8828e-03, -2.2736e-03, -1.0010e-02, -1.0620e-02,
451
+ 2.7618e-03, -1.3184e-02, -1.9775e-02, 2.8198e-02, 3.6469e-03,
452
+ -2.2125e-03, 1.0803e-02, -4.2419e-03, 2.6489e-02, 1.6846e-02,
453
+ 6.4468e-04, -2.1362e-02, -1.1475e-02, 1.4038e-03, -1.4832e-02,
454
+ -1.2024e-02, 3.3203e-02, -1.9775e-02, 1.5831e-04, 1.4572e-03,
455
+ -1.4038e-02, 1.5564e-02, 3.3188e-04, 2.4292e-02, -6.4087e-03,
456
+ -1.5869e-02, 8.1177e-03, -1.1169e-02, 2.6489e-02, -6.7749e-03,
457
+ -7.5378e-03, 3.2227e-02, 2.7588e-02, 3.7079e-03, -1.0071e-02,
458
+ -2.3804e-03, 3.2959e-03, 4.7607e-03, -1.3367e-02, 4.6997e-03,
459
+ 1.0910e-03, -9.7046e-03, 2.0630e-02, -4.3335e-03, -8.9264e-04,
460
+ 6.2866e-03, -2.0264e-02, 2.7771e-03, 1.6724e-02, -1.4771e-02,
461
+ 6.6223e-03, 2.9297e-03, 1.1780e-02, -5.6076e-04, -1.6602e-02,
462
+ -4.9438e-03, -1.1719e-02, -2.7100e-02, -4.1016e-02, -8.7280e-03,
463
+ 1.2085e-02, 1.4465e-02, 1.5259e-03, 2.2949e-02, 7.5989e-03,
464
+ -2.0905e-03, 1.8433e-02, 5.9814e-03, 1.0010e-02, -1.8677e-02,
465
+ 1.2268e-02, 4.7302e-03, 4.3945e-03, 1.4526e-02, -7.4158e-03,
466
+ -2.7008e-03, 9.0942e-03, -3.8300e-03, -1.1414e-02, -1.1742e-05,
467
+ 1.6357e-02, 3.8910e-03, -1.7471e-03, 1.8845e-03, -1.0071e-02,
468
+ 4.4250e-03, 2.5391e-02, 1.1597e-02, 2.5146e-02, 6.8665e-03,
469
+ 1.9531e-02, -6.8359e-03, -1.2817e-02, -7.4768e-03, -1.5747e-02,
470
+ -2.3937e-04, 2.5635e-03, -1.9775e-02, -2.2461e-02, 2.0508e-02,
471
+ 9.1553e-03, 1.8158e-03, -2.0504e-04, -1.0864e-02, -1.5198e-02,
472
+ -1.3199e-03, 9.5825e-03, -3.0518e-03, 7.7515e-03, 2.0142e-02,
473
+ 1.7212e-02, -4.6158e-04, 9.1553e-03, -6.1951e-03, -5.5542e-03,
474
+ -2.2339e-02, -3.4332e-03, -1.6235e-02, 1.8677e-02, 1.8555e-02,
475
+ 8.7280e-03, 7.8735e-03, 1.1353e-02, -5.4321e-03, 2.0142e-02,
476
+ -8.8501e-03, -8.7280e-03, -1.4893e-02, -6.0120e-03, -1.2970e-04,
477
+ 2.1240e-02, 2.2705e-02, -3.3569e-04, 2.6367e-02, -5.6763e-03,
478
+ 1.2024e-02, -1.8555e-02, 2.1667e-03, -1.0376e-02, -5.4321e-03,
479
+ -1.2695e-02, -1.1719e-02, 1.4038e-02, -2.7084e-04, 4.4250e-03,
480
+ -6.6223e-03, -2.1515e-03, 1.4221e-02, 1.9531e-02, -3.0518e-04,
481
+ -1.9531e-02, 7.8735e-03, 1.6357e-02, 1.5991e-02, -7.2632e-03,
482
+ 2.6093e-03, 1.6479e-02, 8.0566e-03, -9.0942e-03, 8.4229e-03,
483
+ -1.9531e-02, -1.6602e-02, -2.4414e-02, -1.1963e-02, -1.9550e-04,
484
+ 6.7139e-03, -8.7280e-03, -8.6670e-03, -2.9053e-02, -4.8523e-03,
485
+ 7.5989e-03, -9.0942e-03, 5.5847e-03, -1.3794e-02, -1.4526e-02,
486
+ 9.3384e-03, -4.0283e-03, 5.1270e-03, -6.9427e-04, 9.3384e-03,
487
+ -1.3672e-02, 3.9101e-04, 1.1719e-02, -3.0823e-03, 1.1230e-02,
488
+ 3.9368e-03, 1.7212e-02, -1.3504e-03, 5.9509e-03, -9.1171e-04,
489
+ 2.2217e-02, -2.5635e-02, -1.2329e-02, -6.1646e-03, -5.3223e-02,
490
+ 1.4343e-03, 1.5076e-02, 2.7008e-03, -2.6611e-02, -1.0742e-02,
491
+ 1.5869e-02, 4.3640e-03, -5.7983e-03, 3.0884e-02, -5.0049e-03,
492
+ -1.0376e-02, -1.1292e-02, 2.4902e-02, 6.0730e-03, 4.5776e-03,
493
+ -1.8799e-02, -4.6997e-03, -5.5542e-03, 1.2390e-02, 8.7891e-03,
494
+ -7.8735e-03, -2.3193e-02, 7.8125e-03, -2.5513e-02, -1.8555e-02,
495
+ -2.0752e-03, -1.3611e-02, 3.1128e-02, -2.0996e-02, 9.8419e-04,
496
+ 3.1250e-02, -1.2634e-02, -1.6724e-02, 1.1414e-02, -1.3550e-02,
497
+ 1.7944e-02, -1.3733e-02, 2.4414e-02, 1.6968e-02, 1.6235e-02,
498
+ -2.3560e-02, -1.7578e-02, -2.3804e-03, -1.6479e-02, -1.6235e-02,
499
+ 1.8799e-02, 1.1353e-02, -1.0620e-02, 1.7090e-02, 2.7344e-02,
500
+ -1.2939e-02, 7.3853e-03, 1.7212e-02, 1.1536e-02, 1.6708e-03,
501
+ 8.7280e-03, 1.3245e-02, 9.8267e-03, -5.3406e-03, 1.8845e-03,
502
+ 9.5215e-03, -1.2634e-02, -4.0527e-02, 1.4160e-02, -1.1169e-02,
503
+ -2.0020e-02, -1.7822e-02, -7.5989e-03, -3.6621e-03, 1.4191e-03,
504
+ 3.0151e-02, -1.0315e-02, 2.2095e-02, -1.9165e-02, -3.1738e-02,
505
+ -1.3428e-02, 7.5684e-03, 3.3691e-02, 1.3428e-02, -1.1826e-03,
506
+ -3.7384e-03, -1.3672e-02, -4.4556e-03, -2.4536e-02, 8.7280e-03,
507
+ 7.2937e-03, 1.6212e-05, 1.0254e-02, 9.4604e-03, 1.2512e-02,
508
+ -7.0496e-03, -4.0588e-03, -4.4250e-03, -1.5869e-02, 2.8809e-02,
509
+ -2.5368e-04, 1.7090e-02, -9.2163e-03, 9.0332e-03, 1.1597e-02,
510
+ 4.4861e-03, 9.1553e-03, -5.8594e-03, -2.4414e-03, -1.0498e-02,
511
+ -1.6602e-02, -5.8594e-03, -4.6082e-03, -3.9673e-03, 1.1414e-02,
512
+ 3.4637e-03, 2.0020e-02, 1.7212e-02, -1.1414e-02, -2.9297e-03,
513
+ 1.8433e-02, 9.7656e-03, 4.9133e-03, 2.2705e-02, 2.4261e-03,
514
+ -6.1035e-03, 3.4027e-03, 1.7014e-03, -1.0742e-02, -6.5430e-02,
515
+ -1.3306e-02, -2.1484e-02, 1.6968e-02, -9.1553e-03, -5.5542e-03,
516
+ 9.2163e-03, -2.0142e-03, -1.8188e-02, -3.4904e-04, -3.2959e-02,
517
+ -1.9409e-02, 5.9204e-03, 1.6937e-03, -7.9346e-03, -8.9722e-03,
518
+ 2.5635e-03, -9.1553e-03, 1.8555e-02, -4.2725e-03, -3.2715e-02,
519
+ 1.7090e-02, 2.6855e-03, -3.3760e-04, 1.1902e-02, 8.4229e-03,
520
+ -4.4250e-04, -1.0803e-02, 2.2949e-02, 8.9111e-03, 8.4229e-03,
521
+ 1.8799e-02, 1.4954e-02, 5.3406e-03, 6.2561e-03, 6.0120e-03,
522
+ 5.2490e-03, -1.7944e-02, -1.5503e-02, -3.1281e-03, -1.2817e-02,
523
+ -1.9043e-02, 9.8419e-04, -4.3945e-03, 1.9897e-02, 6.0120e-03,
524
+ 5.4199e-02, 1.7853e-03, 6.8665e-04, -1.7334e-02, -2.3193e-02,
525
+ 6.7444e-03, 1.8188e-02, 4.2114e-03, -3.2227e-02, 1.1902e-02,
526
+ 1.5869e-02, -8.7280e-03, 7.5531e-04, 1.4709e-02, 1.0681e-02,
527
+ -5.9814e-03, -6.7444e-03, 1.1414e-02, 1.8555e-02, 9.7046e-03,
528
+ -2.2705e-02, -1.0376e-02, -1.7456e-02, -5.0659e-03, -6.4697e-03,
529
+ -2.0020e-02, 5.1270e-03, 1.7578e-02, -3.7689e-03, 1.5015e-02,
530
+ -4.1504e-03, 2.8320e-02, 1.0803e-02, 2.8229e-03, -2.0752e-02,
531
+ -1.7456e-02, 2.5787e-03, -2.2583e-02, -1.3733e-02, 2.9602e-03,
532
+ 1.6357e-02, -9.0332e-03, 4.6692e-03, -9.7656e-03, 4.6921e-04,
533
+ 5.7373e-03, 1.1475e-02, 3.8528e-04, 1.0803e-02, -6.3477e-03,
534
+ 4.3030e-03, -2.0630e-02, 1.1719e-02, 2.6245e-02, -1.9897e-02,
535
+ 9.8877e-03, -3.5706e-03, 8.7357e-04, -5.7068e-03, -1.1047e-02,
536
+ 2.2888e-03, 4.9438e-03, 2.2888e-03, 4.5471e-03, -2.0752e-03,
537
+ 6.9275e-03, -2.0142e-02, 3.2471e-02, 1.3306e-02, -7.8735e-03,
538
+ -1.7456e-02, 2.1118e-02, 2.6611e-02, 3.4912e-02, 1.0437e-02,
539
+ 2.8687e-03, 2.3438e-02, 2.9175e-02, 2.3560e-02, -4.5166e-03,
540
+ -2.1973e-02, 3.0518e-02, -3.9673e-03, 6.4850e-04, 7.1716e-03,
541
+ -7.8735e-03, -2.6245e-03, 1.9531e-02, -2.4414e-03, 1.6113e-02,
542
+ 3.0029e-02, 4.6997e-03, 5.7983e-03, 5.6763e-03, 3.7231e-03,
543
+ 1.4648e-02, -1.2817e-02, -6.9275e-03, 3.5858e-03, -2.0386e-02,
544
+ 1.9775e-02, 1.3611e-02, 1.6113e-02, -9.2773e-03, -1.5442e-02,
545
+ 1.9775e-02, -2.8381e-03, 1.2695e-02, 1.3306e-02, 2.9449e-03,
546
+ 1.0925e-02, -1.2207e-02, -1.1475e-02, -1.1520e-03, -1.2970e-03,
547
+ 8.3618e-03, 1.3046e-03, -1.3885e-03, -5.0354e-03, -1.0986e-02,
548
+ -2.2827e-02, -1.3367e-02, -3.0029e-02, 1.3657e-03, 1.0376e-02,
549
+ -1.8188e-02, -8.4839e-03, 7.5989e-03, 1.4832e-02, 2.5269e-02,
550
+ -6.0272e-04, 1.4954e-02, 3.9673e-03, 5.0049e-03, 3.0151e-02,
551
+ -7.2327e-03, -1.7944e-02, 2.2217e-02, 5.6152e-03, -1.1215e-03,
552
+ -8.3618e-03, -2.2461e-02, 1.1597e-02, -2.3499e-03, 7.8125e-03,
553
+ 1.5381e-02, -1.1292e-03, 3.0670e-03, 1.9989e-03, 9.5825e-03,
554
+ 5.7068e-03, -8.1787e-03, -2.6978e-02, 8.1787e-03, 1.4420e-03,
555
+ -8.4229e-03, -9.9182e-04, -8.3618e-03, -2.4261e-03, 6.4697e-03,
556
+ 1.5625e-02, 5.7983e-03, -1.6235e-02, 5.0049e-03, 2.7954e-02,
557
+ 1.8677e-02, 3.0762e-02, 1.6785e-03, -1.8555e-02, -1.2146e-02,
558
+ -1.5503e-02, 4.4250e-03, 1.4526e-02, -1.1536e-02, -1.0864e-02,
559
+ 2.0630e-02, -2.4658e-02, -1.5442e-02, -1.2207e-02, -2.3926e-02,
560
+ 8.5449e-03, -1.6846e-02, -5.2795e-03, 1.1353e-02, 5.1270e-03,
561
+ 1.4496e-03, -1.1719e-02, -2.4567e-03, 6.4969e-06, -1.9684e-03,
562
+ -2.3438e-02, 8.3008e-03, -7.9346e-03, -1.5747e-02, 2.5879e-02,
563
+ 2.1210e-03, -5.8289e-03, -4.3945e-03, -5.1270e-03, -3.3569e-03,
564
+ 1.2146e-02, -2.7466e-02, 1.7700e-02, 7.3853e-03, 1.5869e-02,
565
+ -1.1292e-02, 2.4414e-02, -7.1411e-03, -6.7139e-03, -1.5974e-05,
566
+ -1.0681e-02, 1.2207e-02, 5.1575e-03, 1.6479e-03, -1.9989e-03,
567
+ 4.5471e-03, 4.6082e-03, 1.1169e-02, 1.3733e-03, -1.3504e-03,
568
+ -8.4686e-04, 2.6245e-02, -2.0752e-03, 1.1368e-03, -5.3406e-03,
569
+ 7.9956e-03, -1.6357e-02, 6.9580e-03, 1.2756e-02, -2.7344e-02,
570
+ -6.5613e-03, 2.8839e-03, -1.3367e-02, -1.7212e-02, -1.4526e-02,
571
+ -9.2163e-03, 2.0630e-02, 5.4016e-03, 2.2949e-02, 7.9956e-03,
572
+ -3.4180e-03, -2.4780e-02, -3.2654e-03, 1.0147e-03, 1.4221e-02,
573
+ 6.2256e-03, -1.7242e-03, -1.9531e-02, -3.4943e-03, -9.3384e-03,
574
+ -1.5869e-02, 1.1902e-03, -1.1902e-02, -7.6599e-03, -1.8799e-02,
575
+ 3.6774e-03, -6.8970e-03, 1.2939e-02, -4.3945e-03, 3.3203e-02,
576
+ -2.4048e-02, 3.0212e-03, -4.8523e-03, 3.4027e-03, -9.1934e-04,
577
+ 1.3306e-02, 3.2959e-02, -2.1606e-02, -1.8799e-02, -9.6436e-03,
578
+ -1.1047e-02, -8.6060e-03, 1.7334e-02, -1.0193e-02, 8.2397e-03,
579
+ 4.0894e-03, -7.6599e-03, -4.6158e-04, -6.1951e-03, 8.9722e-03,
580
+ 1.7090e-03, 2.5513e-02, 1.1719e-02, -1.1719e-02, 9.3384e-03,
581
+ -7.5989e-03, 1.7578e-02, 5.2795e-03, 1.3306e-02, -1.0986e-02,
582
+ 3.1281e-03, -1.8066e-02, -1.5320e-02, 1.7090e-03, 1.2451e-02,
583
+ 3.2654e-03, 4.2236e-02, 1.4221e-02, -1.5381e-02, 1.6724e-02,
584
+ -2.4536e-02, -2.3651e-03, 3.0060e-03, 8.0566e-03, 1.1963e-02,
585
+ 2.8076e-03, -7.5684e-03, 1.0254e-02, 2.1851e-02, -1.2756e-02,
586
+ 1.5625e-02, 2.8564e-02, -2.9144e-03, -5.4016e-03, 2.1362e-02,
587
+ 9.8877e-03, 1.4954e-02, 2.6855e-02, -1.0559e-02, -1.1536e-02,
588
+ 8.1787e-03, -9.9487e-03, 7.3853e-03, 5.7068e-03, -1.8433e-02,
589
+ 1.1963e-02]]]), tensor([[[ 4.2419e-03, -1.9165e-02, 2.3926e-02, 1.2024e-02, 2.7222e-02,
590
+ 3.0327e-04, 5.7373e-03, 1.2329e-02, -1.8539e-03, 6.2256e-03,
591
+ -1.1520e-03, -1.9409e-02, -4.3030e-03, 2.2339e-02, -2.5024e-03,
592
+ 1.2451e-02, -5.3711e-03, -8.1787e-03, 1.5564e-02, -7.8125e-03,
593
+ -1.0010e-02, 8.5449e-03, 1.7319e-03, 4.0894e-03, -1.7578e-02,
594
+ 1.2131e-03, 3.2959e-03, 1.3000e-02, 4.5776e-03, 1.0559e-02,
595
+ -7.5073e-03, -1.4893e-02, 3.9978e-03, 1.7090e-02, 1.4267e-03,
596
+ -9.6436e-03, -1.3794e-02, -1.4648e-02, 1.7700e-02, 9.6436e-03,
597
+ -5.9204e-03, -3.5156e-02, 1.5137e-02, 1.1169e-02, 3.0273e-02,
598
+ 1.5564e-02, -1.9287e-02, -3.3203e-02, -1.8311e-02, -2.5391e-02,
599
+ -1.0986e-03, -6.5613e-03, 7.5073e-03, 2.2461e-02, 1.1108e-02,
600
+ 1.3184e-02, -4.5166e-03, -1.7090e-02, 9.8419e-04, -1.6235e-02,
601
+ 9.1553e-03, -9.3079e-04, -2.0386e-02, -1.3794e-02, 2.4048e-02,
602
+ -9.4604e-03, -1.8799e-02, 3.1891e-03, 2.8687e-02, 6.4697e-03,
603
+ 5.4932e-03, 1.4038e-02, 1.7456e-02, -1.2817e-02, 1.8188e-02,
604
+ -1.6327e-03, 1.2878e-02, 1.8311e-02, 1.0193e-02, -8.3008e-03,
605
+ -2.1210e-03, -4.7302e-03, 8.6975e-04, 1.7853e-03, -1.5137e-02,
606
+ -6.5918e-03, 1.6357e-02, 1.1780e-02, 1.1047e-02, 2.0752e-02,
607
+ 2.4719e-03, 2.0599e-03, -4.4250e-03, 2.6550e-03, 6.1035e-03,
608
+ 2.8931e-02, 2.5513e-02, -3.2715e-02, -1.5442e-02, 4.7363e-02,
609
+ 1.0452e-03, -1.7334e-02, -5.1117e-04, 1.5259e-02, 5.3711e-03,
610
+ 2.5940e-03, 1.4893e-02, -3.3447e-02, -6.4392e-03, -2.0386e-02,
611
+ 9.0942e-03, 1.8539e-03, 2.0752e-02, -7.6904e-03, -3.0640e-02,
612
+ -9.6436e-03, 1.1658e-02, -3.7842e-03, -7.5378e-03, -4.7302e-03,
613
+ -2.9663e-02, -1.3916e-02, -1.3184e-02, -2.0599e-03, -7.4768e-03,
614
+ 4.6387e-03, -2.0874e-02, -1.5381e-02, -1.1963e-02, -1.7578e-02,
615
+ -2.1484e-02, -1.6724e-02, -1.6602e-02, 1.9531e-02, -1.4587e-02,
616
+ 3.3691e-02, 8.7280e-03, -1.3367e-02, -3.2715e-02, 1.2268e-02,
617
+ 2.8320e-02, -2.3682e-02, 2.9297e-03, -8.1177e-03, 1.8921e-03,
618
+ -1.0452e-03, -1.0437e-02, 1.6479e-02, 4.3030e-03, -1.2589e-03,
619
+ -9.9945e-04, 4.4250e-04, -6.6223e-03, -3.4332e-04, 2.5391e-02,
620
+ 5.4321e-03, -1.8555e-02, 6.0425e-03, -1.4343e-02, 4.9973e-04,
621
+ -2.1973e-03, 3.0670e-03, 1.2207e-02, 6.5613e-03, -6.8665e-03,
622
+ -9.7656e-03, 1.6968e-02, -2.2583e-02, 1.9287e-02, 1.8188e-02,
623
+ 8.6060e-03, -1.0315e-02, 2.1362e-03, -1.2024e-02, -1.3550e-02,
624
+ -1.3000e-02, -2.8809e-02, -2.9144e-03, 2.1729e-02, 3.4180e-03,
625
+ 1.9989e-03, -1.0193e-02, -3.2043e-03, 2.1851e-02, 8.6060e-03,
626
+ -8.9722e-03, -1.6602e-02, -1.6235e-02, 7.8125e-03, 5.1270e-03,
627
+ -1.1292e-02, 1.6113e-02, 1.8188e-02, -1.0071e-02, -1.5320e-02,
628
+ -6.3705e-04, -4.3030e-03, 2.5391e-02, 7.9346e-03, -5.0964e-03,
629
+ -2.4414e-04, -3.5156e-02, -1.9531e-03, 1.6357e-02, -3.7384e-03,
630
+ -1.6594e-04, 2.6093e-03, 3.4790e-03, -1.2451e-02, -2.9297e-02,
631
+ 1.8311e-02, 2.8076e-02, 3.4332e-03, -1.9165e-02, 1.4954e-03,
632
+ -2.3315e-02, -7.7209e-03, 2.3560e-02, 6.5918e-03, -5.9509e-03,
633
+ 2.6123e-02, -8.4839e-03, 7.0496e-03, 1.3123e-02, -6.5308e-03,
634
+ -7.0190e-03, 7.4768e-03, 9.4223e-04, 1.4038e-02, 2.0630e-02,
635
+ -2.6703e-03, -6.7234e-05, 1.3611e-02, -1.5991e-02, 2.1667e-03,
636
+ -1.3184e-02, 1.4771e-02, 7.9956e-03, 1.4404e-02, 1.9653e-02,
637
+ -1.1230e-02, 1.0071e-02, -5.4932e-03, 9.6436e-03, 2.1362e-02,
638
+ 8.3618e-03, 1.0300e-03, 1.5198e-02, -1.5747e-02, 1.0803e-02,
639
+ 2.0264e-02, 4.7302e-04, 6.5918e-03, 1.4526e-02, 9.8877e-03,
640
+ 2.7618e-03, -1.7822e-02, -3.9368e-03, -2.6978e-02, -8.5449e-03,
641
+ -2.8229e-03, 1.8188e-02, 3.6163e-03, -8.0566e-03, -2.9144e-03,
642
+ -2.2217e-02, -1.0376e-02, 1.7944e-02, 1.8311e-04, -1.4099e-02,
643
+ 8.6060e-03, 8.9645e-04, -8.7280e-03, -5.4932e-03, -1.1597e-02,
644
+ 1.3733e-02, -1.2329e-02, -6.4850e-04, 3.1738e-02, -8.4229e-03,
645
+ -1.0742e-02, 1.6479e-02, -3.7109e-02, -1.6602e-02, -1.3245e-02,
646
+ 1.4771e-02, -3.8757e-03, 3.0396e-02, -8.7280e-03, 4.7112e-04,
647
+ 1.4648e-02, -2.3071e-02, 1.0681e-02, -1.2573e-02, 1.1047e-02,
648
+ -1.4954e-02, 1.6846e-02, 5.8365e-04, 7.4863e-05, 1.9653e-02,
649
+ 2.6123e-02, 1.7700e-02, 6.3782e-03, 2.0874e-02, 1.5411e-03,
650
+ -5.9204e-03, -1.3245e-02, -2.5024e-03, 3.2196e-03, -6.5308e-03,
651
+ -1.8555e-02, 9.9182e-04, -1.6235e-02, 6.9885e-03, -1.7944e-02,
652
+ -1.1414e-02, 1.9989e-03, -1.9165e-02, -6.4392e-03, 7.4463e-03,
653
+ 8.7891e-03, 9.8877e-03, -7.3242e-04, -1.4343e-02, -6.5613e-03,
654
+ 1.7456e-02, 1.3000e-02, -1.7822e-02, -7.9346e-03, -1.9287e-02,
655
+ 1.6357e-02, -2.0020e-02, -8.1787e-03, 1.3657e-03, 8.8501e-03,
656
+ 7.2937e-03, -1.2939e-02, 1.0071e-02, -1.0376e-02, -1.0498e-02,
657
+ -4.7119e-02, 3.9795e-02, -1.9165e-02, 3.1128e-02, 5.2185e-03,
658
+ 2.0142e-02, -6.4392e-03, -2.4414e-03, 3.1494e-02, 1.6846e-02,
659
+ 8.7891e-03, 5.0964e-03, -8.2397e-03, -3.1494e-02, -6.2866e-03,
660
+ -1.2512e-02, 1.5335e-03, 9.0332e-03, -1.9287e-02, -4.9133e-03,
661
+ 2.3804e-02, 9.7656e-03, 1.5991e-02, -9.3994e-03, -1.0193e-02,
662
+ 8.3618e-03, -7.7515e-03, -2.1606e-02, 1.1963e-02, -2.4658e-02,
663
+ 4.1425e-06, 8.9111e-03, -9.0942e-03, -3.4027e-03, 9.0027e-04,
664
+ -2.3438e-02, 5.6152e-03, 2.1667e-03, 1.4648e-02, 1.3245e-02,
665
+ -1.7700e-02, 6.6833e-03, 5.4016e-03, -1.9379e-03, -3.0396e-02,
666
+ -9.0027e-04, 1.1292e-02, -2.3651e-03, 5.1498e-04, 1.0681e-02,
667
+ 1.0559e-02, -3.7689e-03, -2.4170e-02, -1.1292e-02, -1.0132e-02,
668
+ -1.8677e-02, 2.1362e-02, 2.9785e-02, 2.2125e-03, -3.9978e-03,
669
+ 6.5002e-03, 7.3853e-03, -2.2736e-03, -2.7008e-03, -3.0212e-03,
670
+ -1.8677e-02, -3.2471e-02, -1.0315e-02, -1.9165e-02, -3.7689e-03,
671
+ 3.5858e-03, -9.3384e-03, -1.0620e-02, 1.9775e-02, -2.5024e-03,
672
+ -2.8839e-03, -1.5869e-02, 1.6968e-02, -1.3550e-02, -4.5776e-03,
673
+ -1.2939e-02, 2.6733e-02, 4.8218e-03, 5.4321e-03, 4.7913e-03,
674
+ 3.5286e-04, -1.3245e-02, 5.1880e-03, 3.7956e-04, 2.9663e-02,
675
+ -6.5002e-03, -1.2573e-02, -7.7820e-03, 1.0071e-02, 2.7100e-02,
676
+ -2.6733e-02, 2.9419e-02, -2.5635e-03, 1.2146e-02, -2.2095e-02,
677
+ -2.8076e-03, 9.6436e-03, -7.1716e-03, 1.3794e-02, 1.1108e-02,
678
+ 6.4697e-03, -7.1716e-03, -4.0283e-03, 1.4893e-02, -2.6855e-02,
679
+ -4.3945e-03, -2.3127e-05, 2.0508e-02, -4.4556e-03, -9.8877e-03,
680
+ -1.2390e-02, 5.6152e-03, 7.6599e-03, 1.1292e-02, -1.0498e-02,
681
+ 1.3672e-02, -2.2583e-02, 2.0020e-02, 2.3071e-02, -1.7090e-02,
682
+ 6.8054e-03, 4.8523e-03, -1.8188e-02, -7.4463e-03, -3.7109e-02,
683
+ -5.4321e-03, -2.4536e-02, 1.7456e-02, 3.1738e-02, -1.6968e-02,
684
+ 3.6316e-03, 1.7212e-02, -1.8677e-02, -8.6975e-04, -4.9133e-03,
685
+ 4.6387e-03, -1.7090e-02, 1.1826e-03, 2.5879e-02, -1.2695e-02,
686
+ -4.4861e-03, -1.9379e-03, -1.8406e-04, 2.1729e-02, -8.0109e-04,
687
+ 3.4790e-03, -2.7313e-03, 4.9133e-03, 1.1658e-02, 1.8311e-02,
688
+ 2.2705e-02, -7.1106e-03, 7.7820e-03, -9.0332e-03, -1.4893e-02,
689
+ -7.0572e-04, 3.3691e-02, -8.5449e-03, -1.0925e-02, 7.2937e-03,
690
+ 1.9653e-02, 5.5542e-03, 6.7139e-03, -9.0942e-03, 1.5991e-02,
691
+ -1.2939e-02, -3.4790e-03, -2.6123e-02, -1.0376e-02, -5.6152e-03,
692
+ -2.5391e-02, -3.6049e-04, -1.5869e-02, 3.4485e-03, -2.1667e-03,
693
+ -2.3193e-03, 1.8311e-02, -5.3711e-03, 3.1982e-02, 4.0894e-03,
694
+ -2.0264e-02, -5.4626e-03, -1.3062e-02, 1.6357e-02, 1.1719e-02,
695
+ 3.7231e-03, -1.9165e-02, 2.0752e-03, 3.2425e-04, 9.8877e-03,
696
+ -9.7656e-03, -1.4725e-03, 1.3611e-02, -6.7444e-03, 2.1973e-02,
697
+ 1.7090e-02, 1.2329e-02, -9.5825e-03, 1.8799e-02, 3.5858e-03,
698
+ -1.2573e-02, 6.9885e-03, 2.3804e-02, -2.3315e-02, 7.9956e-03,
699
+ 1.9836e-03, 6.8283e-04, 4.8218e-03, 1.9897e-02, 3.1128e-03,
700
+ 2.4719e-03, 3.0151e-02, -1.5198e-02, 5.2185e-03, 1.5076e-02,
701
+ -1.4648e-02, -1.2146e-02, 1.6235e-02, -5.3406e-03, -1.1749e-03,
702
+ 2.3438e-02, -3.4180e-03, -1.0742e-02, -8.6670e-03, 3.4027e-03,
703
+ -4.1992e-02, -2.1648e-04, -1.1536e-02, 7.5684e-03, 8.9645e-04,
704
+ -2.2217e-02, 7.0496e-03, 6.8359e-03, -1.9531e-02, -8.9722e-03,
705
+ 9.2773e-03, 5.7068e-03, 1.5015e-02, -1.0132e-02, -1.5076e-02,
706
+ -8.5449e-03, 3.2501e-03, -2.2827e-02, -4.2725e-03, 2.0020e-02,
707
+ 9.8877e-03, -1.6357e-02, -7.9346e-03, -5.5237e-03, -2.3041e-03,
708
+ 1.3733e-02, 1.0620e-02, 1.3123e-02, 1.8311e-02, -5.9814e-03,
709
+ 1.8921e-02, -1.6846e-02, 2.6398e-03, 2.3560e-02, -1.3367e-02,
710
+ 6.7139e-03, 1.1780e-02, 1.6724e-02, 6.0120e-03, -1.4954e-02,
711
+ -9.0942e-03, 2.3651e-03, 4.3945e-03, -9.4604e-03, 6.6223e-03,
712
+ -9.2163e-03, 8.4229e-03, -2.1362e-02, 9.5825e-03, -5.2490e-03,
713
+ -1.5564e-02, 1.8387e-03, -5.7373e-03, 1.8555e-02, -9.0942e-03,
714
+ -1.6113e-02, -2.0508e-02, -1.9775e-02, -1.2207e-02, 3.2715e-02,
715
+ 1.3367e-02, -6.5918e-03, 4.6387e-03, 6.6833e-03, -2.8076e-02,
716
+ 1.3306e-02, 1.0757e-03, -3.5156e-02, -1.1780e-02, 1.0315e-02,
717
+ -1.5442e-02, -5.3101e-03, -2.6489e-02, -1.8234e-03, -2.0630e-02,
718
+ 2.0508e-02, -3.6316e-03, -2.0996e-02, -3.0975e-03, -1.3733e-02,
719
+ 2.7954e-02, -2.0630e-02, 2.0905e-03, 2.8076e-03, 1.3794e-02,
720
+ -1.1841e-02, 7.0801e-03, -1.6937e-03, -3.6926e-03, -2.5940e-03,
721
+ -1.0498e-02, -2.8534e-03, 4.6692e-03, -9.1553e-03, -1.3550e-02,
722
+ -2.6550e-03, 7.8735e-03, -1.2878e-02, -2.5879e-02, -9.8877e-03,
723
+ 1.3275e-03, -3.4904e-04, -6.8359e-03, 1.9653e-02, -1.0071e-02,
724
+ 4.1504e-03, -1.5564e-02, -2.1362e-02, 2.3560e-02, -5.0049e-03,
725
+ 1.4282e-02, 4.9133e-03, -1.2054e-03, -8.6060e-03, 1.6724e-02,
726
+ -1.1353e-02, 2.2583e-02, 2.9449e-03, -3.6316e-03, -1.5991e-02,
727
+ 1.4954e-02, 5.1880e-03, 2.3193e-02, 1.4648e-02, 2.2583e-03,
728
+ 1.1536e-02, -6.0654e-04, 6.6833e-03, -5.5542e-03, 2.2583e-03,
729
+ -9.8877e-03, -1.5564e-02, -3.3112e-03, 6.8054e-03, -1.9165e-02,
730
+ 3.4809e-05, -7.5684e-03, 1.8921e-02, 7.5684e-03, -4.3488e-04,
731
+ 9.3994e-03, 2.1667e-03, 2.2583e-02, -4.0771e-02, -2.2278e-03,
732
+ -1.9043e-02, -9.8267e-03, 3.6774e-03, 5.7983e-03, -1.4404e-02,
733
+ -3.4485e-03, -1.0132e-02, 2.1973e-03, -1.2817e-03, 2.6733e-02,
734
+ 6.0272e-04, 3.8330e-02, 1.3062e-02, 1.2817e-02, 2.1484e-02,
735
+ -1.5259e-02, -2.6611e-02, -5.4016e-03, -5.2795e-03, -7.7438e-04,
736
+ -1.5991e-02, 1.5137e-02, -1.1597e-02, 2.2583e-02, -3.3722e-03,
737
+ -4.6539e-04, -1.1414e-02, -2.4170e-02, 8.5449e-03, 2.3560e-02,
738
+ -5.6458e-03, 4.3030e-03, -2.9297e-02, -1.5015e-02, 1.5442e-02,
739
+ -2.0020e-02, -1.2695e-02, 4.0894e-03, -9.8267e-03, 2.2461e-02,
740
+ 1.6235e-02, 1.7700e-02, -2.6733e-02, 1.5259e-02, -7.9956e-03,
741
+ 1.0910e-03, -6.1951e-03, 6.1035e-03, -1.4160e-02, -2.4048e-02,
742
+ -1.4465e-02, -9.3994e-03, 1.0437e-02, 9.0942e-03, -1.6724e-02,
743
+ 7.2937e-03, 2.1362e-02, 3.7079e-03, -3.9368e-03, 1.3123e-03,
744
+ 8.5449e-03, 1.4160e-02, 1.9897e-02, 6.8054e-03, -2.0020e-02,
745
+ -2.2583e-02, -1.6724e-02, 8.0490e-04, 4.0039e-02, 5.7678e-03,
746
+ 1.0620e-02, -1.3733e-02, 1.2878e-02, 1.6602e-02, 7.3547e-03,
747
+ 2.8534e-03, 7.0572e-04, -2.7344e-02, 2.6550e-03, 4.4861e-03,
748
+ -1.3062e-02, 1.4587e-02, -7.7820e-03, 2.8372e-05, -4.6997e-03,
749
+ 1.1169e-02, 3.7689e-03, -1.8311e-02, 6.9885e-03, 1.6708e-03,
750
+ -1.9226e-03, -1.5442e-02, 5.4626e-03, -3.0396e-02, -2.4902e-02,
751
+ 6.3477e-03, -5.7678e-03, -2.4414e-02, -2.5757e-02, 1.4832e-02,
752
+ 2.0020e-02, 1.4648e-02, -5.1117e-04, -5.4016e-03, 1.7334e-02,
753
+ 2.6367e-02, -2.4719e-03, 1.5503e-02, -1.0498e-02, -6.2866e-03,
754
+ 1.8997e-03, -1.9897e-02, 3.5095e-03, 1.4343e-02, 3.1433e-03,
755
+ 2.4902e-02, 4.9438e-03, -1.9775e-02, -2.7100e-02, 8.9111e-03,
756
+ -6.1951e-03, -2.4536e-02, 1.8555e-02, 3.0670e-03, -1.0376e-02,
757
+ 2.4048e-02, 1.0742e-02, -6.8359e-03, 9.2773e-03, -2.1729e-02,
758
+ -1.6113e-02, 3.5248e-03, 3.7079e-03, 3.5095e-03, -3.6774e-03,
759
+ 1.4893e-02, -2.5757e-02, -1.2695e-02, -2.3682e-02, -3.8574e-02,
760
+ 1.7090e-02, -9.6436e-03, -2.1729e-02, -6.0425e-03, -1.5381e-02,
761
+ -1.8311e-02, -2.1515e-03, 1.3123e-02, 2.8076e-02, -1.2207e-02,
762
+ 5.6076e-04, -3.0518e-03, 8.8501e-03, 6.1035e-03, -1.0193e-02,
763
+ 2.2583e-02, 6.0425e-03, -1.5015e-02, 5.4626e-03, -8.0566e-03,
764
+ 2.2217e-02, 9.0942e-03, -2.9449e-03, 1.2589e-03, 7.6294e-03,
765
+ -1.8799e-02, -4.2969e-02, 5.6152e-03, 1.9287e-02, -3.5553e-03,
766
+ 4.2725e-03, -3.5889e-02, 3.3569e-03, -6.0120e-03, 2.6398e-03,
767
+ -1.4893e-02, -4.7302e-03, 1.0864e-02, -2.3956e-03, 2.8931e-02,
768
+ 1.1292e-02]]]), tensor([[[-0.0250, -0.0245, 0.0175, ..., 0.0332, 0.0106, 0.0041],
769
+ [ 0.0060, -0.0280, 0.0079, ..., -0.0086, 0.0168, 0.0135],
770
+ [ 0.0121, 0.0053, -0.0028, ..., 0.0192, 0.0135, 0.0059],
771
+ ...,
772
+ [ 0.0245, 0.0073, 0.0044, ..., 0.0104, -0.0150, -0.0141],
773
+ [ 0.0060, -0.0280, 0.0079, ..., -0.0086, 0.0168, 0.0135],
774
+ [ 0.0123, -0.0060, 0.0310, ..., 0.0021, 0.0104, 0.0239]]],
775
+ grad_fn=<CatBackward0>)]
776
+
777
+ Generaded and Prepared the inputs:
778
+ tensor([[[-0.0020, 0.0210, -0.0137, ..., 0.0057, -0.0184, 0.0120],
779
+ [-0.0020, 0.0210, -0.0137, ..., 0.0057, -0.0184, 0.0120],
780
+ [-0.0120, -0.0040, 0.0083, ..., 0.0083, 0.0131, -0.0119],
781
+ ...,
782
+ [ 0.0123, -0.0060, 0.0310, ..., 0.0021, 0.0104, 0.0239],
783
+ [-0.0020, 0.0210, -0.0137, ..., 0.0057, -0.0184, 0.0120],
784
+ [-0.0030, -0.0425, 0.0232, ..., 0.0081, 0.0537, -0.0090]]],
785
+ grad_fn=<CatBackward0>)
786
+ Generates IDs from the Reasoner:
787
+ tensor([[ 362, 883, 374, 11699, 389, 264, 13425, 11, 9963, 264,
788
+ 3460, 91613, 315, 19281, 13, 576, 6249, 15562, 82, 304,
789
+ 389, 279, 883, 594, 3579, 323, 279, 91613, 438, 566,
790
+ 9982, 432, 705, 13, 576, 5112, 6239, 369, 19281, 323,
791
+ 7203, 525, 59021, 11, 18860, 429, 419, 6109, 374, 4363]])
792
+ Tokenized the Outputs
793
+ A man is sitting on a bench, holding a large bouquet of flowers. The camera zooms in on the man's face and the bouquet as he holds it up. The sound effects for flowers and movement are muted, indicating that this scene is likely
model_architecture.txt ADDED
@@ -0,0 +1,1977 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Currently not using the OG trained model for easy and fast loading, ...
2
+ Used LLM Qwen2 0.5B
3
+
4
+ JOSIE(
5
+ (imagebind_encoder): ImageBindModel(
6
+ (modality_preprocessors): ModuleDict(
7
+ (vision): RGBDTPreprocessor(
8
+ (cls_token): tensor((1, 1, 1280), requires_grad=False)
9
+
10
+ (rgbt_stem): PatchEmbedGeneric(
11
+ (proj): Sequential(
12
+ (0): PadIm2Video()
13
+ (1): Conv3d(3, 1280, kernel_size=(2, 14, 14), stride=(2, 14, 14), bias=False)
14
+ )
15
+ )
16
+ (pos_embedding_helper): SpatioTemporalPosEmbeddingHelper(
17
+ (pos_embed): tensor((1, 257, 1280), requires_grad=False)
18
+
19
+ )
20
+ )
21
+ (text): TextPreprocessor(
22
+ (pos_embed): tensor((1, 77, 1024), requires_grad=False)
23
+ (mask): tensor((77, 77), requires_grad=False)
24
+
25
+ (token_embedding): Embedding(49408, 1024)
26
+ )
27
+ (audio): AudioPreprocessor(
28
+ (cls_token): tensor((1, 1, 768), requires_grad=False)
29
+
30
+ (rgbt_stem): PatchEmbedGeneric(
31
+ (proj): Conv2d(1, 768, kernel_size=(16, 16), stride=(10, 10), bias=False)
32
+ (norm_layer): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
33
+ )
34
+ (pos_embedding_helper): SpatioTemporalPosEmbeddingHelper(
35
+ (pos_embed): tensor((1, 229, 768), requires_grad=False)
36
+
37
+ )
38
+ )
39
+ (depth): RGBDTPreprocessor(
40
+ (cls_token): tensor((1, 1, 384), requires_grad=False)
41
+
42
+ (depth_stem): PatchEmbedGeneric(
43
+ (proj): Conv2d(1, 384, kernel_size=(16, 16), stride=(16, 16), bias=False)
44
+ (norm_layer): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
45
+ )
46
+ (pos_embedding_helper): SpatioTemporalPosEmbeddingHelper(
47
+ (pos_embed): tensor((1, 197, 384), requires_grad=False)
48
+
49
+ )
50
+ )
51
+ (thermal): ThermalPreprocessor(
52
+ (cls_token): tensor((1, 1, 768), requires_grad=False)
53
+
54
+ (rgbt_stem): PatchEmbedGeneric(
55
+ (proj): Conv2d(1, 768, kernel_size=(16, 16), stride=(16, 16), bias=False)
56
+ (norm_layer): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
57
+ )
58
+ (pos_embedding_helper): SpatioTemporalPosEmbeddingHelper(
59
+ (pos_embed): tensor((1, 197, 768), requires_grad=False)
60
+
61
+ )
62
+ )
63
+ (imu): IMUPreprocessor(
64
+ (pos_embed): tensor((1, 251, 512), requires_grad=False)
65
+ (cls_token): tensor((1, 1, 512), requires_grad=False)
66
+
67
+ (imu_stem): PatchEmbedGeneric(
68
+ (proj): Linear(in_features=48, out_features=512, bias=False)
69
+ (norm_layer): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
70
+ )
71
+ )
72
+ )
73
+ (modality_trunks): ModuleDict(
74
+ (vision): SimpleTransformer(
75
+ (pre_transformer_layer): Sequential(
76
+ (0): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
77
+ (1): EinOpsRearrange()
78
+ )
79
+ (blocks): Sequential(
80
+ (0): BlockWithMasking(
81
+ (attn): MultiheadAttention(
82
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
83
+ )
84
+ (drop_path): Identity()
85
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
86
+ (mlp): Mlp(
87
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
88
+ (act): GELU(approximate='none')
89
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
90
+ (drop): Dropout(p=0.0, inplace=False)
91
+ )
92
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
93
+ )
94
+ (1): BlockWithMasking(
95
+ (attn): MultiheadAttention(
96
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
97
+ )
98
+ (drop_path): Identity()
99
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
100
+ (mlp): Mlp(
101
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
102
+ (act): GELU(approximate='none')
103
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
104
+ (drop): Dropout(p=0.0, inplace=False)
105
+ )
106
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
107
+ )
108
+ (2): BlockWithMasking(
109
+ (attn): MultiheadAttention(
110
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
111
+ )
112
+ (drop_path): Identity()
113
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
114
+ (mlp): Mlp(
115
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
116
+ (act): GELU(approximate='none')
117
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
118
+ (drop): Dropout(p=0.0, inplace=False)
119
+ )
120
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
121
+ )
122
+ (3): BlockWithMasking(
123
+ (attn): MultiheadAttention(
124
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
125
+ )
126
+ (drop_path): Identity()
127
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
128
+ (mlp): Mlp(
129
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
130
+ (act): GELU(approximate='none')
131
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
132
+ (drop): Dropout(p=0.0, inplace=False)
133
+ )
134
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
135
+ )
136
+ (4): BlockWithMasking(
137
+ (attn): MultiheadAttention(
138
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
139
+ )
140
+ (drop_path): Identity()
141
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
142
+ (mlp): Mlp(
143
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
144
+ (act): GELU(approximate='none')
145
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
146
+ (drop): Dropout(p=0.0, inplace=False)
147
+ )
148
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
149
+ )
150
+ (5): BlockWithMasking(
151
+ (attn): MultiheadAttention(
152
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
153
+ )
154
+ (drop_path): Identity()
155
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
156
+ (mlp): Mlp(
157
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
158
+ (act): GELU(approximate='none')
159
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
160
+ (drop): Dropout(p=0.0, inplace=False)
161
+ )
162
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
163
+ )
164
+ (6): BlockWithMasking(
165
+ (attn): MultiheadAttention(
166
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
167
+ )
168
+ (drop_path): Identity()
169
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
170
+ (mlp): Mlp(
171
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
172
+ (act): GELU(approximate='none')
173
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
174
+ (drop): Dropout(p=0.0, inplace=False)
175
+ )
176
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
177
+ )
178
+ (7): BlockWithMasking(
179
+ (attn): MultiheadAttention(
180
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
181
+ )
182
+ (drop_path): Identity()
183
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
184
+ (mlp): Mlp(
185
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
186
+ (act): GELU(approximate='none')
187
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
188
+ (drop): Dropout(p=0.0, inplace=False)
189
+ )
190
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
191
+ )
192
+ (8): BlockWithMasking(
193
+ (attn): MultiheadAttention(
194
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
195
+ )
196
+ (drop_path): Identity()
197
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
198
+ (mlp): Mlp(
199
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
200
+ (act): GELU(approximate='none')
201
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
202
+ (drop): Dropout(p=0.0, inplace=False)
203
+ )
204
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
205
+ )
206
+ (9): BlockWithMasking(
207
+ (attn): MultiheadAttention(
208
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
209
+ )
210
+ (drop_path): Identity()
211
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
212
+ (mlp): Mlp(
213
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
214
+ (act): GELU(approximate='none')
215
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
216
+ (drop): Dropout(p=0.0, inplace=False)
217
+ )
218
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
219
+ )
220
+ (10): BlockWithMasking(
221
+ (attn): MultiheadAttention(
222
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
223
+ )
224
+ (drop_path): Identity()
225
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
226
+ (mlp): Mlp(
227
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
228
+ (act): GELU(approximate='none')
229
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
230
+ (drop): Dropout(p=0.0, inplace=False)
231
+ )
232
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
233
+ )
234
+ (11): BlockWithMasking(
235
+ (attn): MultiheadAttention(
236
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
237
+ )
238
+ (drop_path): Identity()
239
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
240
+ (mlp): Mlp(
241
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
242
+ (act): GELU(approximate='none')
243
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
244
+ (drop): Dropout(p=0.0, inplace=False)
245
+ )
246
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
247
+ )
248
+ (12): BlockWithMasking(
249
+ (attn): MultiheadAttention(
250
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
251
+ )
252
+ (drop_path): Identity()
253
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
254
+ (mlp): Mlp(
255
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
256
+ (act): GELU(approximate='none')
257
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
258
+ (drop): Dropout(p=0.0, inplace=False)
259
+ )
260
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
261
+ )
262
+ (13): BlockWithMasking(
263
+ (attn): MultiheadAttention(
264
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
265
+ )
266
+ (drop_path): Identity()
267
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
268
+ (mlp): Mlp(
269
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
270
+ (act): GELU(approximate='none')
271
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
272
+ (drop): Dropout(p=0.0, inplace=False)
273
+ )
274
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
275
+ )
276
+ (14): BlockWithMasking(
277
+ (attn): MultiheadAttention(
278
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
279
+ )
280
+ (drop_path): Identity()
281
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
282
+ (mlp): Mlp(
283
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
284
+ (act): GELU(approximate='none')
285
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
286
+ (drop): Dropout(p=0.0, inplace=False)
287
+ )
288
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
289
+ )
290
+ (15): BlockWithMasking(
291
+ (attn): MultiheadAttention(
292
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
293
+ )
294
+ (drop_path): Identity()
295
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
296
+ (mlp): Mlp(
297
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
298
+ (act): GELU(approximate='none')
299
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
300
+ (drop): Dropout(p=0.0, inplace=False)
301
+ )
302
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
303
+ )
304
+ (16): BlockWithMasking(
305
+ (attn): MultiheadAttention(
306
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
307
+ )
308
+ (drop_path): Identity()
309
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
310
+ (mlp): Mlp(
311
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
312
+ (act): GELU(approximate='none')
313
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
314
+ (drop): Dropout(p=0.0, inplace=False)
315
+ )
316
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
317
+ )
318
+ (17): BlockWithMasking(
319
+ (attn): MultiheadAttention(
320
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
321
+ )
322
+ (drop_path): Identity()
323
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
324
+ (mlp): Mlp(
325
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
326
+ (act): GELU(approximate='none')
327
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
328
+ (drop): Dropout(p=0.0, inplace=False)
329
+ )
330
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
331
+ )
332
+ (18): BlockWithMasking(
333
+ (attn): MultiheadAttention(
334
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
335
+ )
336
+ (drop_path): Identity()
337
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
338
+ (mlp): Mlp(
339
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
340
+ (act): GELU(approximate='none')
341
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
342
+ (drop): Dropout(p=0.0, inplace=False)
343
+ )
344
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
345
+ )
346
+ (19): BlockWithMasking(
347
+ (attn): MultiheadAttention(
348
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
349
+ )
350
+ (drop_path): Identity()
351
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
352
+ (mlp): Mlp(
353
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
354
+ (act): GELU(approximate='none')
355
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
356
+ (drop): Dropout(p=0.0, inplace=False)
357
+ )
358
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
359
+ )
360
+ (20): BlockWithMasking(
361
+ (attn): MultiheadAttention(
362
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
363
+ )
364
+ (drop_path): Identity()
365
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
366
+ (mlp): Mlp(
367
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
368
+ (act): GELU(approximate='none')
369
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
370
+ (drop): Dropout(p=0.0, inplace=False)
371
+ )
372
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
373
+ )
374
+ (21): BlockWithMasking(
375
+ (attn): MultiheadAttention(
376
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
377
+ )
378
+ (drop_path): Identity()
379
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
380
+ (mlp): Mlp(
381
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
382
+ (act): GELU(approximate='none')
383
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
384
+ (drop): Dropout(p=0.0, inplace=False)
385
+ )
386
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
387
+ )
388
+ (22): BlockWithMasking(
389
+ (attn): MultiheadAttention(
390
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
391
+ )
392
+ (drop_path): Identity()
393
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
394
+ (mlp): Mlp(
395
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
396
+ (act): GELU(approximate='none')
397
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
398
+ (drop): Dropout(p=0.0, inplace=False)
399
+ )
400
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
401
+ )
402
+ (23): BlockWithMasking(
403
+ (attn): MultiheadAttention(
404
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
405
+ )
406
+ (drop_path): Identity()
407
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
408
+ (mlp): Mlp(
409
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
410
+ (act): GELU(approximate='none')
411
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
412
+ (drop): Dropout(p=0.0, inplace=False)
413
+ )
414
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
415
+ )
416
+ (24): BlockWithMasking(
417
+ (attn): MultiheadAttention(
418
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
419
+ )
420
+ (drop_path): Identity()
421
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
422
+ (mlp): Mlp(
423
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
424
+ (act): GELU(approximate='none')
425
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
426
+ (drop): Dropout(p=0.0, inplace=False)
427
+ )
428
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
429
+ )
430
+ (25): BlockWithMasking(
431
+ (attn): MultiheadAttention(
432
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
433
+ )
434
+ (drop_path): Identity()
435
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
436
+ (mlp): Mlp(
437
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
438
+ (act): GELU(approximate='none')
439
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
440
+ (drop): Dropout(p=0.0, inplace=False)
441
+ )
442
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
443
+ )
444
+ (26): BlockWithMasking(
445
+ (attn): MultiheadAttention(
446
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
447
+ )
448
+ (drop_path): Identity()
449
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
450
+ (mlp): Mlp(
451
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
452
+ (act): GELU(approximate='none')
453
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
454
+ (drop): Dropout(p=0.0, inplace=False)
455
+ )
456
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
457
+ )
458
+ (27): BlockWithMasking(
459
+ (attn): MultiheadAttention(
460
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
461
+ )
462
+ (drop_path): Identity()
463
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
464
+ (mlp): Mlp(
465
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
466
+ (act): GELU(approximate='none')
467
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
468
+ (drop): Dropout(p=0.0, inplace=False)
469
+ )
470
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
471
+ )
472
+ (28): BlockWithMasking(
473
+ (attn): MultiheadAttention(
474
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
475
+ )
476
+ (drop_path): Identity()
477
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
478
+ (mlp): Mlp(
479
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
480
+ (act): GELU(approximate='none')
481
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
482
+ (drop): Dropout(p=0.0, inplace=False)
483
+ )
484
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
485
+ )
486
+ (29): BlockWithMasking(
487
+ (attn): MultiheadAttention(
488
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
489
+ )
490
+ (drop_path): Identity()
491
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
492
+ (mlp): Mlp(
493
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
494
+ (act): GELU(approximate='none')
495
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
496
+ (drop): Dropout(p=0.0, inplace=False)
497
+ )
498
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
499
+ )
500
+ (30): BlockWithMasking(
501
+ (attn): MultiheadAttention(
502
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
503
+ )
504
+ (drop_path): Identity()
505
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
506
+ (mlp): Mlp(
507
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
508
+ (act): GELU(approximate='none')
509
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
510
+ (drop): Dropout(p=0.0, inplace=False)
511
+ )
512
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
513
+ )
514
+ (31): BlockWithMasking(
515
+ (attn): MultiheadAttention(
516
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1280, out_features=1280, bias=True)
517
+ )
518
+ (drop_path): Identity()
519
+ (norm_1): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
520
+ (mlp): Mlp(
521
+ (fc1): Linear(in_features=1280, out_features=5120, bias=True)
522
+ (act): GELU(approximate='none')
523
+ (fc2): Linear(in_features=5120, out_features=1280, bias=True)
524
+ (drop): Dropout(p=0.0, inplace=False)
525
+ )
526
+ (norm_2): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
527
+ )
528
+ )
529
+ (post_transformer_layer): EinOpsRearrange()
530
+ )
531
+ (text): SimpleTransformer(
532
+ (pre_transformer_layer): Sequential(
533
+ (0): Identity()
534
+ (1): EinOpsRearrange()
535
+ )
536
+ (blocks): Sequential(
537
+ (0): BlockWithMasking(
538
+ (attn): MultiheadAttention(
539
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
540
+ )
541
+ (drop_path): Identity()
542
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
543
+ (mlp): Mlp(
544
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
545
+ (act): GELU(approximate='none')
546
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
547
+ (drop): Dropout(p=0.0, inplace=False)
548
+ )
549
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
550
+ )
551
+ (1): BlockWithMasking(
552
+ (attn): MultiheadAttention(
553
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
554
+ )
555
+ (drop_path): Identity()
556
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
557
+ (mlp): Mlp(
558
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
559
+ (act): GELU(approximate='none')
560
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
561
+ (drop): Dropout(p=0.0, inplace=False)
562
+ )
563
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
564
+ )
565
+ (2): BlockWithMasking(
566
+ (attn): MultiheadAttention(
567
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
568
+ )
569
+ (drop_path): Identity()
570
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
571
+ (mlp): Mlp(
572
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
573
+ (act): GELU(approximate='none')
574
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
575
+ (drop): Dropout(p=0.0, inplace=False)
576
+ )
577
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
578
+ )
579
+ (3): BlockWithMasking(
580
+ (attn): MultiheadAttention(
581
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
582
+ )
583
+ (drop_path): Identity()
584
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
585
+ (mlp): Mlp(
586
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
587
+ (act): GELU(approximate='none')
588
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
589
+ (drop): Dropout(p=0.0, inplace=False)
590
+ )
591
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
592
+ )
593
+ (4): BlockWithMasking(
594
+ (attn): MultiheadAttention(
595
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
596
+ )
597
+ (drop_path): Identity()
598
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
599
+ (mlp): Mlp(
600
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
601
+ (act): GELU(approximate='none')
602
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
603
+ (drop): Dropout(p=0.0, inplace=False)
604
+ )
605
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
606
+ )
607
+ (5): BlockWithMasking(
608
+ (attn): MultiheadAttention(
609
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
610
+ )
611
+ (drop_path): Identity()
612
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
613
+ (mlp): Mlp(
614
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
615
+ (act): GELU(approximate='none')
616
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
617
+ (drop): Dropout(p=0.0, inplace=False)
618
+ )
619
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
620
+ )
621
+ (6): BlockWithMasking(
622
+ (attn): MultiheadAttention(
623
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
624
+ )
625
+ (drop_path): Identity()
626
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
627
+ (mlp): Mlp(
628
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
629
+ (act): GELU(approximate='none')
630
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
631
+ (drop): Dropout(p=0.0, inplace=False)
632
+ )
633
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
634
+ )
635
+ (7): BlockWithMasking(
636
+ (attn): MultiheadAttention(
637
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
638
+ )
639
+ (drop_path): Identity()
640
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
641
+ (mlp): Mlp(
642
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
643
+ (act): GELU(approximate='none')
644
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
645
+ (drop): Dropout(p=0.0, inplace=False)
646
+ )
647
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
648
+ )
649
+ (8): BlockWithMasking(
650
+ (attn): MultiheadAttention(
651
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
652
+ )
653
+ (drop_path): Identity()
654
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
655
+ (mlp): Mlp(
656
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
657
+ (act): GELU(approximate='none')
658
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
659
+ (drop): Dropout(p=0.0, inplace=False)
660
+ )
661
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
662
+ )
663
+ (9): BlockWithMasking(
664
+ (attn): MultiheadAttention(
665
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
666
+ )
667
+ (drop_path): Identity()
668
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
669
+ (mlp): Mlp(
670
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
671
+ (act): GELU(approximate='none')
672
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
673
+ (drop): Dropout(p=0.0, inplace=False)
674
+ )
675
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
676
+ )
677
+ (10): BlockWithMasking(
678
+ (attn): MultiheadAttention(
679
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
680
+ )
681
+ (drop_path): Identity()
682
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
683
+ (mlp): Mlp(
684
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
685
+ (act): GELU(approximate='none')
686
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
687
+ (drop): Dropout(p=0.0, inplace=False)
688
+ )
689
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
690
+ )
691
+ (11): BlockWithMasking(
692
+ (attn): MultiheadAttention(
693
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
694
+ )
695
+ (drop_path): Identity()
696
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
697
+ (mlp): Mlp(
698
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
699
+ (act): GELU(approximate='none')
700
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
701
+ (drop): Dropout(p=0.0, inplace=False)
702
+ )
703
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
704
+ )
705
+ (12): BlockWithMasking(
706
+ (attn): MultiheadAttention(
707
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
708
+ )
709
+ (drop_path): Identity()
710
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
711
+ (mlp): Mlp(
712
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
713
+ (act): GELU(approximate='none')
714
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
715
+ (drop): Dropout(p=0.0, inplace=False)
716
+ )
717
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
718
+ )
719
+ (13): BlockWithMasking(
720
+ (attn): MultiheadAttention(
721
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
722
+ )
723
+ (drop_path): Identity()
724
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
725
+ (mlp): Mlp(
726
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
727
+ (act): GELU(approximate='none')
728
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
729
+ (drop): Dropout(p=0.0, inplace=False)
730
+ )
731
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
732
+ )
733
+ (14): BlockWithMasking(
734
+ (attn): MultiheadAttention(
735
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
736
+ )
737
+ (drop_path): Identity()
738
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
739
+ (mlp): Mlp(
740
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
741
+ (act): GELU(approximate='none')
742
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
743
+ (drop): Dropout(p=0.0, inplace=False)
744
+ )
745
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
746
+ )
747
+ (15): BlockWithMasking(
748
+ (attn): MultiheadAttention(
749
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
750
+ )
751
+ (drop_path): Identity()
752
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
753
+ (mlp): Mlp(
754
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
755
+ (act): GELU(approximate='none')
756
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
757
+ (drop): Dropout(p=0.0, inplace=False)
758
+ )
759
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
760
+ )
761
+ (16): BlockWithMasking(
762
+ (attn): MultiheadAttention(
763
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
764
+ )
765
+ (drop_path): Identity()
766
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
767
+ (mlp): Mlp(
768
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
769
+ (act): GELU(approximate='none')
770
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
771
+ (drop): Dropout(p=0.0, inplace=False)
772
+ )
773
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
774
+ )
775
+ (17): BlockWithMasking(
776
+ (attn): MultiheadAttention(
777
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
778
+ )
779
+ (drop_path): Identity()
780
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
781
+ (mlp): Mlp(
782
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
783
+ (act): GELU(approximate='none')
784
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
785
+ (drop): Dropout(p=0.0, inplace=False)
786
+ )
787
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
788
+ )
789
+ (18): BlockWithMasking(
790
+ (attn): MultiheadAttention(
791
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
792
+ )
793
+ (drop_path): Identity()
794
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
795
+ (mlp): Mlp(
796
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
797
+ (act): GELU(approximate='none')
798
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
799
+ (drop): Dropout(p=0.0, inplace=False)
800
+ )
801
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
802
+ )
803
+ (19): BlockWithMasking(
804
+ (attn): MultiheadAttention(
805
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
806
+ )
807
+ (drop_path): Identity()
808
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
809
+ (mlp): Mlp(
810
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
811
+ (act): GELU(approximate='none')
812
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
813
+ (drop): Dropout(p=0.0, inplace=False)
814
+ )
815
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
816
+ )
817
+ (20): BlockWithMasking(
818
+ (attn): MultiheadAttention(
819
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
820
+ )
821
+ (drop_path): Identity()
822
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
823
+ (mlp): Mlp(
824
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
825
+ (act): GELU(approximate='none')
826
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
827
+ (drop): Dropout(p=0.0, inplace=False)
828
+ )
829
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
830
+ )
831
+ (21): BlockWithMasking(
832
+ (attn): MultiheadAttention(
833
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
834
+ )
835
+ (drop_path): Identity()
836
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
837
+ (mlp): Mlp(
838
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
839
+ (act): GELU(approximate='none')
840
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
841
+ (drop): Dropout(p=0.0, inplace=False)
842
+ )
843
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
844
+ )
845
+ (22): BlockWithMasking(
846
+ (attn): MultiheadAttention(
847
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
848
+ )
849
+ (drop_path): Identity()
850
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
851
+ (mlp): Mlp(
852
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
853
+ (act): GELU(approximate='none')
854
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
855
+ (drop): Dropout(p=0.0, inplace=False)
856
+ )
857
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
858
+ )
859
+ (23): BlockWithMasking(
860
+ (attn): MultiheadAttention(
861
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=1024, out_features=1024, bias=True)
862
+ )
863
+ (drop_path): Identity()
864
+ (norm_1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
865
+ (mlp): Mlp(
866
+ (fc1): Linear(in_features=1024, out_features=4096, bias=True)
867
+ (act): GELU(approximate='none')
868
+ (fc2): Linear(in_features=4096, out_features=1024, bias=True)
869
+ (drop): Dropout(p=0.0, inplace=False)
870
+ )
871
+ (norm_2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
872
+ )
873
+ )
874
+ (post_transformer_layer): EinOpsRearrange()
875
+ )
876
+ (audio): SimpleTransformer(
877
+ (pre_transformer_layer): Sequential(
878
+ (0): Identity()
879
+ (1): EinOpsRearrange()
880
+ )
881
+ (blocks): Sequential(
882
+ (0): BlockWithMasking(
883
+ (attn): MultiheadAttention(
884
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
885
+ )
886
+ (drop_path): Identity()
887
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
888
+ (mlp): Mlp(
889
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
890
+ (act): GELU(approximate='none')
891
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
892
+ (drop): Dropout(p=0.0, inplace=False)
893
+ )
894
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
895
+ )
896
+ (1): BlockWithMasking(
897
+ (attn): MultiheadAttention(
898
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
899
+ )
900
+ (drop_path): DropPath(drop_prob=0.009)
901
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
902
+ (mlp): Mlp(
903
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
904
+ (act): GELU(approximate='none')
905
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
906
+ (drop): Dropout(p=0.0, inplace=False)
907
+ )
908
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
909
+ )
910
+ (2): BlockWithMasking(
911
+ (attn): MultiheadAttention(
912
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
913
+ )
914
+ (drop_path): DropPath(drop_prob=0.018)
915
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
916
+ (mlp): Mlp(
917
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
918
+ (act): GELU(approximate='none')
919
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
920
+ (drop): Dropout(p=0.0, inplace=False)
921
+ )
922
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
923
+ )
924
+ (3): BlockWithMasking(
925
+ (attn): MultiheadAttention(
926
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
927
+ )
928
+ (drop_path): DropPath(drop_prob=0.027)
929
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
930
+ (mlp): Mlp(
931
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
932
+ (act): GELU(approximate='none')
933
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
934
+ (drop): Dropout(p=0.0, inplace=False)
935
+ )
936
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
937
+ )
938
+ (4): BlockWithMasking(
939
+ (attn): MultiheadAttention(
940
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
941
+ )
942
+ (drop_path): DropPath(drop_prob=0.036)
943
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
944
+ (mlp): Mlp(
945
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
946
+ (act): GELU(approximate='none')
947
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
948
+ (drop): Dropout(p=0.0, inplace=False)
949
+ )
950
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
951
+ )
952
+ (5): BlockWithMasking(
953
+ (attn): MultiheadAttention(
954
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
955
+ )
956
+ (drop_path): DropPath(drop_prob=0.045)
957
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
958
+ (mlp): Mlp(
959
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
960
+ (act): GELU(approximate='none')
961
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
962
+ (drop): Dropout(p=0.0, inplace=False)
963
+ )
964
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
965
+ )
966
+ (6): BlockWithMasking(
967
+ (attn): MultiheadAttention(
968
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
969
+ )
970
+ (drop_path): DropPath(drop_prob=0.055)
971
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
972
+ (mlp): Mlp(
973
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
974
+ (act): GELU(approximate='none')
975
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
976
+ (drop): Dropout(p=0.0, inplace=False)
977
+ )
978
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
979
+ )
980
+ (7): BlockWithMasking(
981
+ (attn): MultiheadAttention(
982
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
983
+ )
984
+ (drop_path): DropPath(drop_prob=0.064)
985
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
986
+ (mlp): Mlp(
987
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
988
+ (act): GELU(approximate='none')
989
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
990
+ (drop): Dropout(p=0.0, inplace=False)
991
+ )
992
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
993
+ )
994
+ (8): BlockWithMasking(
995
+ (attn): MultiheadAttention(
996
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
997
+ )
998
+ (drop_path): DropPath(drop_prob=0.073)
999
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1000
+ (mlp): Mlp(
1001
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1002
+ (act): GELU(approximate='none')
1003
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1004
+ (drop): Dropout(p=0.0, inplace=False)
1005
+ )
1006
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1007
+ )
1008
+ (9): BlockWithMasking(
1009
+ (attn): MultiheadAttention(
1010
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1011
+ )
1012
+ (drop_path): DropPath(drop_prob=0.082)
1013
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1014
+ (mlp): Mlp(
1015
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1016
+ (act): GELU(approximate='none')
1017
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1018
+ (drop): Dropout(p=0.0, inplace=False)
1019
+ )
1020
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1021
+ )
1022
+ (10): BlockWithMasking(
1023
+ (attn): MultiheadAttention(
1024
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1025
+ )
1026
+ (drop_path): DropPath(drop_prob=0.091)
1027
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1028
+ (mlp): Mlp(
1029
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1030
+ (act): GELU(approximate='none')
1031
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1032
+ (drop): Dropout(p=0.0, inplace=False)
1033
+ )
1034
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1035
+ )
1036
+ (11): BlockWithMasking(
1037
+ (attn): MultiheadAttention(
1038
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1039
+ )
1040
+ (drop_path): DropPath(drop_prob=0.100)
1041
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1042
+ (mlp): Mlp(
1043
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1044
+ (act): GELU(approximate='none')
1045
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1046
+ (drop): Dropout(p=0.0, inplace=False)
1047
+ )
1048
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1049
+ )
1050
+ )
1051
+ (post_transformer_layer): EinOpsRearrange()
1052
+ )
1053
+ (depth): SimpleTransformer(
1054
+ (pre_transformer_layer): Sequential(
1055
+ (0): Identity()
1056
+ (1): EinOpsRearrange()
1057
+ )
1058
+ (blocks): Sequential(
1059
+ (0): BlockWithMasking(
1060
+ (attn): MultiheadAttention(
1061
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1062
+ )
1063
+ (drop_path): Identity()
1064
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1065
+ (mlp): Mlp(
1066
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1067
+ (act): GELU(approximate='none')
1068
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1069
+ (drop): Dropout(p=0.0, inplace=False)
1070
+ )
1071
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1072
+ )
1073
+ (1): BlockWithMasking(
1074
+ (attn): MultiheadAttention(
1075
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1076
+ )
1077
+ (drop_path): Identity()
1078
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1079
+ (mlp): Mlp(
1080
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1081
+ (act): GELU(approximate='none')
1082
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1083
+ (drop): Dropout(p=0.0, inplace=False)
1084
+ )
1085
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1086
+ )
1087
+ (2): BlockWithMasking(
1088
+ (attn): MultiheadAttention(
1089
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1090
+ )
1091
+ (drop_path): Identity()
1092
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1093
+ (mlp): Mlp(
1094
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1095
+ (act): GELU(approximate='none')
1096
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1097
+ (drop): Dropout(p=0.0, inplace=False)
1098
+ )
1099
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1100
+ )
1101
+ (3): BlockWithMasking(
1102
+ (attn): MultiheadAttention(
1103
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1104
+ )
1105
+ (drop_path): Identity()
1106
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1107
+ (mlp): Mlp(
1108
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1109
+ (act): GELU(approximate='none')
1110
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1111
+ (drop): Dropout(p=0.0, inplace=False)
1112
+ )
1113
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1114
+ )
1115
+ (4): BlockWithMasking(
1116
+ (attn): MultiheadAttention(
1117
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1118
+ )
1119
+ (drop_path): Identity()
1120
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1121
+ (mlp): Mlp(
1122
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1123
+ (act): GELU(approximate='none')
1124
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1125
+ (drop): Dropout(p=0.0, inplace=False)
1126
+ )
1127
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1128
+ )
1129
+ (5): BlockWithMasking(
1130
+ (attn): MultiheadAttention(
1131
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1132
+ )
1133
+ (drop_path): Identity()
1134
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1135
+ (mlp): Mlp(
1136
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1137
+ (act): GELU(approximate='none')
1138
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1139
+ (drop): Dropout(p=0.0, inplace=False)
1140
+ )
1141
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1142
+ )
1143
+ (6): BlockWithMasking(
1144
+ (attn): MultiheadAttention(
1145
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1146
+ )
1147
+ (drop_path): Identity()
1148
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1149
+ (mlp): Mlp(
1150
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1151
+ (act): GELU(approximate='none')
1152
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1153
+ (drop): Dropout(p=0.0, inplace=False)
1154
+ )
1155
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1156
+ )
1157
+ (7): BlockWithMasking(
1158
+ (attn): MultiheadAttention(
1159
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1160
+ )
1161
+ (drop_path): Identity()
1162
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1163
+ (mlp): Mlp(
1164
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1165
+ (act): GELU(approximate='none')
1166
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1167
+ (drop): Dropout(p=0.0, inplace=False)
1168
+ )
1169
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1170
+ )
1171
+ (8): BlockWithMasking(
1172
+ (attn): MultiheadAttention(
1173
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1174
+ )
1175
+ (drop_path): Identity()
1176
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1177
+ (mlp): Mlp(
1178
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1179
+ (act): GELU(approximate='none')
1180
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1181
+ (drop): Dropout(p=0.0, inplace=False)
1182
+ )
1183
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1184
+ )
1185
+ (9): BlockWithMasking(
1186
+ (attn): MultiheadAttention(
1187
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1188
+ )
1189
+ (drop_path): Identity()
1190
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1191
+ (mlp): Mlp(
1192
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1193
+ (act): GELU(approximate='none')
1194
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1195
+ (drop): Dropout(p=0.0, inplace=False)
1196
+ )
1197
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1198
+ )
1199
+ (10): BlockWithMasking(
1200
+ (attn): MultiheadAttention(
1201
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1202
+ )
1203
+ (drop_path): Identity()
1204
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1205
+ (mlp): Mlp(
1206
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1207
+ (act): GELU(approximate='none')
1208
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1209
+ (drop): Dropout(p=0.0, inplace=False)
1210
+ )
1211
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1212
+ )
1213
+ (11): BlockWithMasking(
1214
+ (attn): MultiheadAttention(
1215
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=384, out_features=384, bias=True)
1216
+ )
1217
+ (drop_path): Identity()
1218
+ (norm_1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1219
+ (mlp): Mlp(
1220
+ (fc1): Linear(in_features=384, out_features=1536, bias=True)
1221
+ (act): GELU(approximate='none')
1222
+ (fc2): Linear(in_features=1536, out_features=384, bias=True)
1223
+ (drop): Dropout(p=0.0, inplace=False)
1224
+ )
1225
+ (norm_2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1226
+ )
1227
+ )
1228
+ (post_transformer_layer): EinOpsRearrange()
1229
+ )
1230
+ (thermal): SimpleTransformer(
1231
+ (pre_transformer_layer): Sequential(
1232
+ (0): Identity()
1233
+ (1): EinOpsRearrange()
1234
+ )
1235
+ (blocks): Sequential(
1236
+ (0): BlockWithMasking(
1237
+ (attn): MultiheadAttention(
1238
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1239
+ )
1240
+ (drop_path): Identity()
1241
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1242
+ (mlp): Mlp(
1243
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1244
+ (act): GELU(approximate='none')
1245
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1246
+ (drop): Dropout(p=0.0, inplace=False)
1247
+ )
1248
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1249
+ )
1250
+ (1): BlockWithMasking(
1251
+ (attn): MultiheadAttention(
1252
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1253
+ )
1254
+ (drop_path): Identity()
1255
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1256
+ (mlp): Mlp(
1257
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1258
+ (act): GELU(approximate='none')
1259
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1260
+ (drop): Dropout(p=0.0, inplace=False)
1261
+ )
1262
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1263
+ )
1264
+ (2): BlockWithMasking(
1265
+ (attn): MultiheadAttention(
1266
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1267
+ )
1268
+ (drop_path): Identity()
1269
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1270
+ (mlp): Mlp(
1271
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1272
+ (act): GELU(approximate='none')
1273
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1274
+ (drop): Dropout(p=0.0, inplace=False)
1275
+ )
1276
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1277
+ )
1278
+ (3): BlockWithMasking(
1279
+ (attn): MultiheadAttention(
1280
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1281
+ )
1282
+ (drop_path): Identity()
1283
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1284
+ (mlp): Mlp(
1285
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1286
+ (act): GELU(approximate='none')
1287
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1288
+ (drop): Dropout(p=0.0, inplace=False)
1289
+ )
1290
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1291
+ )
1292
+ (4): BlockWithMasking(
1293
+ (attn): MultiheadAttention(
1294
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1295
+ )
1296
+ (drop_path): Identity()
1297
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1298
+ (mlp): Mlp(
1299
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1300
+ (act): GELU(approximate='none')
1301
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1302
+ (drop): Dropout(p=0.0, inplace=False)
1303
+ )
1304
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1305
+ )
1306
+ (5): BlockWithMasking(
1307
+ (attn): MultiheadAttention(
1308
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1309
+ )
1310
+ (drop_path): Identity()
1311
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1312
+ (mlp): Mlp(
1313
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1314
+ (act): GELU(approximate='none')
1315
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1316
+ (drop): Dropout(p=0.0, inplace=False)
1317
+ )
1318
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1319
+ )
1320
+ (6): BlockWithMasking(
1321
+ (attn): MultiheadAttention(
1322
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1323
+ )
1324
+ (drop_path): Identity()
1325
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1326
+ (mlp): Mlp(
1327
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1328
+ (act): GELU(approximate='none')
1329
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1330
+ (drop): Dropout(p=0.0, inplace=False)
1331
+ )
1332
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1333
+ )
1334
+ (7): BlockWithMasking(
1335
+ (attn): MultiheadAttention(
1336
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1337
+ )
1338
+ (drop_path): Identity()
1339
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1340
+ (mlp): Mlp(
1341
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1342
+ (act): GELU(approximate='none')
1343
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1344
+ (drop): Dropout(p=0.0, inplace=False)
1345
+ )
1346
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1347
+ )
1348
+ (8): BlockWithMasking(
1349
+ (attn): MultiheadAttention(
1350
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1351
+ )
1352
+ (drop_path): Identity()
1353
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1354
+ (mlp): Mlp(
1355
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1356
+ (act): GELU(approximate='none')
1357
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1358
+ (drop): Dropout(p=0.0, inplace=False)
1359
+ )
1360
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1361
+ )
1362
+ (9): BlockWithMasking(
1363
+ (attn): MultiheadAttention(
1364
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1365
+ )
1366
+ (drop_path): Identity()
1367
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1368
+ (mlp): Mlp(
1369
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1370
+ (act): GELU(approximate='none')
1371
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1372
+ (drop): Dropout(p=0.0, inplace=False)
1373
+ )
1374
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1375
+ )
1376
+ (10): BlockWithMasking(
1377
+ (attn): MultiheadAttention(
1378
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1379
+ )
1380
+ (drop_path): Identity()
1381
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1382
+ (mlp): Mlp(
1383
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1384
+ (act): GELU(approximate='none')
1385
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1386
+ (drop): Dropout(p=0.0, inplace=False)
1387
+ )
1388
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1389
+ )
1390
+ (11): BlockWithMasking(
1391
+ (attn): MultiheadAttention(
1392
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
1393
+ )
1394
+ (drop_path): Identity()
1395
+ (norm_1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1396
+ (mlp): Mlp(
1397
+ (fc1): Linear(in_features=768, out_features=3072, bias=True)
1398
+ (act): GELU(approximate='none')
1399
+ (fc2): Linear(in_features=3072, out_features=768, bias=True)
1400
+ (drop): Dropout(p=0.0, inplace=False)
1401
+ )
1402
+ (norm_2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1403
+ )
1404
+ )
1405
+ (post_transformer_layer): EinOpsRearrange()
1406
+ )
1407
+ (imu): SimpleTransformer(
1408
+ (pre_transformer_layer): Sequential(
1409
+ (0): Identity()
1410
+ (1): EinOpsRearrange()
1411
+ )
1412
+ (blocks): Sequential(
1413
+ (0): BlockWithMasking(
1414
+ (attn): MultiheadAttention(
1415
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
1416
+ )
1417
+ (drop_path): Identity()
1418
+ (norm_1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1419
+ (mlp): Mlp(
1420
+ (fc1): Linear(in_features=512, out_features=2048, bias=True)
1421
+ (act): GELU(approximate='none')
1422
+ (fc2): Linear(in_features=2048, out_features=512, bias=True)
1423
+ (drop): Dropout(p=0.0, inplace=False)
1424
+ )
1425
+ (norm_2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1426
+ )
1427
+ (1): BlockWithMasking(
1428
+ (attn): MultiheadAttention(
1429
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
1430
+ )
1431
+ (drop_path): DropPath(drop_prob=0.140)
1432
+ (norm_1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1433
+ (mlp): Mlp(
1434
+ (fc1): Linear(in_features=512, out_features=2048, bias=True)
1435
+ (act): GELU(approximate='none')
1436
+ (fc2): Linear(in_features=2048, out_features=512, bias=True)
1437
+ (drop): Dropout(p=0.0, inplace=False)
1438
+ )
1439
+ (norm_2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1440
+ )
1441
+ (2): BlockWithMasking(
1442
+ (attn): MultiheadAttention(
1443
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
1444
+ )
1445
+ (drop_path): DropPath(drop_prob=0.280)
1446
+ (norm_1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1447
+ (mlp): Mlp(
1448
+ (fc1): Linear(in_features=512, out_features=2048, bias=True)
1449
+ (act): GELU(approximate='none')
1450
+ (fc2): Linear(in_features=2048, out_features=512, bias=True)
1451
+ (drop): Dropout(p=0.0, inplace=False)
1452
+ )
1453
+ (norm_2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1454
+ )
1455
+ (3): BlockWithMasking(
1456
+ (attn): MultiheadAttention(
1457
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
1458
+ )
1459
+ (drop_path): DropPath(drop_prob=0.420)
1460
+ (norm_1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1461
+ (mlp): Mlp(
1462
+ (fc1): Linear(in_features=512, out_features=2048, bias=True)
1463
+ (act): GELU(approximate='none')
1464
+ (fc2): Linear(in_features=2048, out_features=512, bias=True)
1465
+ (drop): Dropout(p=0.0, inplace=False)
1466
+ )
1467
+ (norm_2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1468
+ )
1469
+ (4): BlockWithMasking(
1470
+ (attn): MultiheadAttention(
1471
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
1472
+ )
1473
+ (drop_path): DropPath(drop_prob=0.560)
1474
+ (norm_1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1475
+ (mlp): Mlp(
1476
+ (fc1): Linear(in_features=512, out_features=2048, bias=True)
1477
+ (act): GELU(approximate='none')
1478
+ (fc2): Linear(in_features=2048, out_features=512, bias=True)
1479
+ (drop): Dropout(p=0.0, inplace=False)
1480
+ )
1481
+ (norm_2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1482
+ )
1483
+ (5): BlockWithMasking(
1484
+ (attn): MultiheadAttention(
1485
+ (out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
1486
+ )
1487
+ (drop_path): DropPath(drop_prob=0.700)
1488
+ (norm_1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1489
+ (mlp): Mlp(
1490
+ (fc1): Linear(in_features=512, out_features=2048, bias=True)
1491
+ (act): GELU(approximate='none')
1492
+ (fc2): Linear(in_features=2048, out_features=512, bias=True)
1493
+ (drop): Dropout(p=0.0, inplace=False)
1494
+ )
1495
+ (norm_2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1496
+ )
1497
+ )
1498
+ (post_transformer_layer): EinOpsRearrange()
1499
+ )
1500
+ )
1501
+ (modality_heads): ModuleDict(
1502
+ (vision): Sequential(
1503
+ (0): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
1504
+ (1): SelectElement()
1505
+ (2): Linear(in_features=1280, out_features=1024, bias=False)
1506
+ )
1507
+ (text): SelectEOSAndProject(
1508
+ (proj): Sequential(
1509
+ (0): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
1510
+ (1): Linear(in_features=1024, out_features=1024, bias=False)
1511
+ )
1512
+ )
1513
+ (audio): Sequential(
1514
+ (0): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1515
+ (1): SelectElement()
1516
+ (2): Linear(in_features=768, out_features=1024, bias=False)
1517
+ )
1518
+ (depth): Sequential(
1519
+ (0): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
1520
+ (1): SelectElement()
1521
+ (2): Linear(in_features=384, out_features=1024, bias=False)
1522
+ )
1523
+ (thermal): Sequential(
1524
+ (0): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
1525
+ (1): SelectElement()
1526
+ (2): Linear(in_features=768, out_features=1024, bias=False)
1527
+ )
1528
+ (imu): Sequential(
1529
+ (0): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
1530
+ (1): SelectElement()
1531
+ (2): Dropout(p=0.5, inplace=False)
1532
+ (3): Linear(in_features=512, out_features=1024, bias=False)
1533
+ )
1534
+ )
1535
+ (modality_postprocessors): ModuleDict(
1536
+ (vision): Normalize()
1537
+ (text): Sequential(
1538
+ (0): Normalize()
1539
+ (1): LearnableLogitScaling(logit_scale_init=14.285714285714285,learnable=True, max_logit_scale=100)
1540
+ )
1541
+ (audio): Sequential(
1542
+ (0): Normalize()
1543
+ (1): LearnableLogitScaling(logit_scale_init=20.0,learnable=False, max_logit_scale=100)
1544
+ )
1545
+ (depth): Sequential(
1546
+ (0): Normalize()
1547
+ (1): LearnableLogitScaling(logit_scale_init=5.0,learnable=False, max_logit_scale=100)
1548
+ )
1549
+ (thermal): Sequential(
1550
+ (0): Normalize()
1551
+ (1): LearnableLogitScaling(logit_scale_init=10.0,learnable=False, max_logit_scale=100)
1552
+ )
1553
+ (imu): Sequential(
1554
+ (0): Normalize()
1555
+ (1): LearnableLogitScaling(logit_scale_init=5.0,learnable=False, max_logit_scale=100)
1556
+ )
1557
+ )
1558
+ )
1559
+ (reasoner): Qwen2ForCausalLM(
1560
+ (model): Qwen2Model(
1561
+ (embed_tokens): Embedding(151936, 896)
1562
+ (layers): ModuleList(
1563
+ (0): Qwen2DecoderLayer(
1564
+ (self_attn): Qwen2Attention(
1565
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1566
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1567
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1568
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1569
+ (rotary_emb): Qwen2RotaryEmbedding()
1570
+ )
1571
+ (mlp): Qwen2MLP(
1572
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1573
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1574
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1575
+ (act_fn): SiLU()
1576
+ )
1577
+ (input_layernorm): Qwen2RMSNorm()
1578
+ (post_attention_layernorm): Qwen2RMSNorm()
1579
+ )
1580
+ (1): Qwen2DecoderLayer(
1581
+ (self_attn): Qwen2Attention(
1582
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1583
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1584
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1585
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1586
+ (rotary_emb): Qwen2RotaryEmbedding()
1587
+ )
1588
+ (mlp): Qwen2MLP(
1589
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1590
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1591
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1592
+ (act_fn): SiLU()
1593
+ )
1594
+ (input_layernorm): Qwen2RMSNorm()
1595
+ (post_attention_layernorm): Qwen2RMSNorm()
1596
+ )
1597
+ (2): Qwen2DecoderLayer(
1598
+ (self_attn): Qwen2Attention(
1599
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1600
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1601
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1602
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1603
+ (rotary_emb): Qwen2RotaryEmbedding()
1604
+ )
1605
+ (mlp): Qwen2MLP(
1606
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1607
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1608
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1609
+ (act_fn): SiLU()
1610
+ )
1611
+ (input_layernorm): Qwen2RMSNorm()
1612
+ (post_attention_layernorm): Qwen2RMSNorm()
1613
+ )
1614
+ (3): Qwen2DecoderLayer(
1615
+ (self_attn): Qwen2Attention(
1616
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1617
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1618
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1619
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1620
+ (rotary_emb): Qwen2RotaryEmbedding()
1621
+ )
1622
+ (mlp): Qwen2MLP(
1623
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1624
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1625
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1626
+ (act_fn): SiLU()
1627
+ )
1628
+ (input_layernorm): Qwen2RMSNorm()
1629
+ (post_attention_layernorm): Qwen2RMSNorm()
1630
+ )
1631
+ (4): Qwen2DecoderLayer(
1632
+ (self_attn): Qwen2Attention(
1633
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1634
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1635
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1636
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1637
+ (rotary_emb): Qwen2RotaryEmbedding()
1638
+ )
1639
+ (mlp): Qwen2MLP(
1640
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1641
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1642
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1643
+ (act_fn): SiLU()
1644
+ )
1645
+ (input_layernorm): Qwen2RMSNorm()
1646
+ (post_attention_layernorm): Qwen2RMSNorm()
1647
+ )
1648
+ (5): Qwen2DecoderLayer(
1649
+ (self_attn): Qwen2Attention(
1650
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1651
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1652
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1653
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1654
+ (rotary_emb): Qwen2RotaryEmbedding()
1655
+ )
1656
+ (mlp): Qwen2MLP(
1657
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1658
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1659
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1660
+ (act_fn): SiLU()
1661
+ )
1662
+ (input_layernorm): Qwen2RMSNorm()
1663
+ (post_attention_layernorm): Qwen2RMSNorm()
1664
+ )
1665
+ (6): Qwen2DecoderLayer(
1666
+ (self_attn): Qwen2Attention(
1667
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1668
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1669
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1670
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1671
+ (rotary_emb): Qwen2RotaryEmbedding()
1672
+ )
1673
+ (mlp): Qwen2MLP(
1674
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1675
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1676
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1677
+ (act_fn): SiLU()
1678
+ )
1679
+ (input_layernorm): Qwen2RMSNorm()
1680
+ (post_attention_layernorm): Qwen2RMSNorm()
1681
+ )
1682
+ (7): Qwen2DecoderLayer(
1683
+ (self_attn): Qwen2Attention(
1684
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1685
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1686
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1687
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1688
+ (rotary_emb): Qwen2RotaryEmbedding()
1689
+ )
1690
+ (mlp): Qwen2MLP(
1691
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1692
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1693
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1694
+ (act_fn): SiLU()
1695
+ )
1696
+ (input_layernorm): Qwen2RMSNorm()
1697
+ (post_attention_layernorm): Qwen2RMSNorm()
1698
+ )
1699
+ (8): Qwen2DecoderLayer(
1700
+ (self_attn): Qwen2Attention(
1701
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1702
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1703
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1704
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1705
+ (rotary_emb): Qwen2RotaryEmbedding()
1706
+ )
1707
+ (mlp): Qwen2MLP(
1708
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1709
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1710
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1711
+ (act_fn): SiLU()
1712
+ )
1713
+ (input_layernorm): Qwen2RMSNorm()
1714
+ (post_attention_layernorm): Qwen2RMSNorm()
1715
+ )
1716
+ (9): Qwen2DecoderLayer(
1717
+ (self_attn): Qwen2Attention(
1718
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1719
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1720
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1721
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1722
+ (rotary_emb): Qwen2RotaryEmbedding()
1723
+ )
1724
+ (mlp): Qwen2MLP(
1725
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1726
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1727
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1728
+ (act_fn): SiLU()
1729
+ )
1730
+ (input_layernorm): Qwen2RMSNorm()
1731
+ (post_attention_layernorm): Qwen2RMSNorm()
1732
+ )
1733
+ (10): Qwen2DecoderLayer(
1734
+ (self_attn): Qwen2Attention(
1735
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1736
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1737
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1738
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1739
+ (rotary_emb): Qwen2RotaryEmbedding()
1740
+ )
1741
+ (mlp): Qwen2MLP(
1742
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1743
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1744
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1745
+ (act_fn): SiLU()
1746
+ )
1747
+ (input_layernorm): Qwen2RMSNorm()
1748
+ (post_attention_layernorm): Qwen2RMSNorm()
1749
+ )
1750
+ (11): Qwen2DecoderLayer(
1751
+ (self_attn): Qwen2Attention(
1752
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1753
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1754
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1755
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1756
+ (rotary_emb): Qwen2RotaryEmbedding()
1757
+ )
1758
+ (mlp): Qwen2MLP(
1759
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1760
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1761
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1762
+ (act_fn): SiLU()
1763
+ )
1764
+ (input_layernorm): Qwen2RMSNorm()
1765
+ (post_attention_layernorm): Qwen2RMSNorm()
1766
+ )
1767
+ (12): Qwen2DecoderLayer(
1768
+ (self_attn): Qwen2Attention(
1769
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1770
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1771
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1772
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1773
+ (rotary_emb): Qwen2RotaryEmbedding()
1774
+ )
1775
+ (mlp): Qwen2MLP(
1776
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1777
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1778
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1779
+ (act_fn): SiLU()
1780
+ )
1781
+ (input_layernorm): Qwen2RMSNorm()
1782
+ (post_attention_layernorm): Qwen2RMSNorm()
1783
+ )
1784
+ (13): Qwen2DecoderLayer(
1785
+ (self_attn): Qwen2Attention(
1786
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1787
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1788
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1789
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1790
+ (rotary_emb): Qwen2RotaryEmbedding()
1791
+ )
1792
+ (mlp): Qwen2MLP(
1793
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1794
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1795
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1796
+ (act_fn): SiLU()
1797
+ )
1798
+ (input_layernorm): Qwen2RMSNorm()
1799
+ (post_attention_layernorm): Qwen2RMSNorm()
1800
+ )
1801
+ (14): Qwen2DecoderLayer(
1802
+ (self_attn): Qwen2Attention(
1803
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1804
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1805
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1806
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1807
+ (rotary_emb): Qwen2RotaryEmbedding()
1808
+ )
1809
+ (mlp): Qwen2MLP(
1810
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1811
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1812
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1813
+ (act_fn): SiLU()
1814
+ )
1815
+ (input_layernorm): Qwen2RMSNorm()
1816
+ (post_attention_layernorm): Qwen2RMSNorm()
1817
+ )
1818
+ (15): Qwen2DecoderLayer(
1819
+ (self_attn): Qwen2Attention(
1820
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1821
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1822
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1823
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1824
+ (rotary_emb): Qwen2RotaryEmbedding()
1825
+ )
1826
+ (mlp): Qwen2MLP(
1827
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1828
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1829
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1830
+ (act_fn): SiLU()
1831
+ )
1832
+ (input_layernorm): Qwen2RMSNorm()
1833
+ (post_attention_layernorm): Qwen2RMSNorm()
1834
+ )
1835
+ (16): Qwen2DecoderLayer(
1836
+ (self_attn): Qwen2Attention(
1837
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1838
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1839
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1840
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1841
+ (rotary_emb): Qwen2RotaryEmbedding()
1842
+ )
1843
+ (mlp): Qwen2MLP(
1844
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1845
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1846
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1847
+ (act_fn): SiLU()
1848
+ )
1849
+ (input_layernorm): Qwen2RMSNorm()
1850
+ (post_attention_layernorm): Qwen2RMSNorm()
1851
+ )
1852
+ (17): Qwen2DecoderLayer(
1853
+ (self_attn): Qwen2Attention(
1854
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1855
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1856
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1857
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1858
+ (rotary_emb): Qwen2RotaryEmbedding()
1859
+ )
1860
+ (mlp): Qwen2MLP(
1861
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1862
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1863
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1864
+ (act_fn): SiLU()
1865
+ )
1866
+ (input_layernorm): Qwen2RMSNorm()
1867
+ (post_attention_layernorm): Qwen2RMSNorm()
1868
+ )
1869
+ (18): Qwen2DecoderLayer(
1870
+ (self_attn): Qwen2Attention(
1871
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1872
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1873
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1874
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1875
+ (rotary_emb): Qwen2RotaryEmbedding()
1876
+ )
1877
+ (mlp): Qwen2MLP(
1878
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1879
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1880
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1881
+ (act_fn): SiLU()
1882
+ )
1883
+ (input_layernorm): Qwen2RMSNorm()
1884
+ (post_attention_layernorm): Qwen2RMSNorm()
1885
+ )
1886
+ (19): Qwen2DecoderLayer(
1887
+ (self_attn): Qwen2Attention(
1888
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1889
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1890
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1891
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1892
+ (rotary_emb): Qwen2RotaryEmbedding()
1893
+ )
1894
+ (mlp): Qwen2MLP(
1895
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1896
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1897
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1898
+ (act_fn): SiLU()
1899
+ )
1900
+ (input_layernorm): Qwen2RMSNorm()
1901
+ (post_attention_layernorm): Qwen2RMSNorm()
1902
+ )
1903
+ (20): Qwen2DecoderLayer(
1904
+ (self_attn): Qwen2Attention(
1905
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1906
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1907
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1908
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1909
+ (rotary_emb): Qwen2RotaryEmbedding()
1910
+ )
1911
+ (mlp): Qwen2MLP(
1912
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1913
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1914
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1915
+ (act_fn): SiLU()
1916
+ )
1917
+ (input_layernorm): Qwen2RMSNorm()
1918
+ (post_attention_layernorm): Qwen2RMSNorm()
1919
+ )
1920
+ (21): Qwen2DecoderLayer(
1921
+ (self_attn): Qwen2Attention(
1922
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1923
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1924
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1925
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1926
+ (rotary_emb): Qwen2RotaryEmbedding()
1927
+ )
1928
+ (mlp): Qwen2MLP(
1929
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1930
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1931
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1932
+ (act_fn): SiLU()
1933
+ )
1934
+ (input_layernorm): Qwen2RMSNorm()
1935
+ (post_attention_layernorm): Qwen2RMSNorm()
1936
+ )
1937
+ (22): Qwen2DecoderLayer(
1938
+ (self_attn): Qwen2Attention(
1939
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1940
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1941
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1942
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1943
+ (rotary_emb): Qwen2RotaryEmbedding()
1944
+ )
1945
+ (mlp): Qwen2MLP(
1946
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1947
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1948
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1949
+ (act_fn): SiLU()
1950
+ )
1951
+ (input_layernorm): Qwen2RMSNorm()
1952
+ (post_attention_layernorm): Qwen2RMSNorm()
1953
+ )
1954
+ (23): Qwen2DecoderLayer(
1955
+ (self_attn): Qwen2Attention(
1956
+ (q_proj): Linear(in_features=896, out_features=896, bias=True)
1957
+ (k_proj): Linear(in_features=896, out_features=128, bias=True)
1958
+ (v_proj): Linear(in_features=896, out_features=128, bias=True)
1959
+ (o_proj): Linear(in_features=896, out_features=896, bias=False)
1960
+ (rotary_emb): Qwen2RotaryEmbedding()
1961
+ )
1962
+ (mlp): Qwen2MLP(
1963
+ (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
1964
+ (up_proj): Linear(in_features=896, out_features=4864, bias=False)
1965
+ (down_proj): Linear(in_features=4864, out_features=896, bias=False)
1966
+ (act_fn): SiLU()
1967
+ )
1968
+ (input_layernorm): Qwen2RMSNorm()
1969
+ (post_attention_layernorm): Qwen2RMSNorm()
1970
+ )
1971
+ )
1972
+ (norm): Qwen2RMSNorm()
1973
+ )
1974
+ (lm_head): Linear(in_features=896, out_features=151936, bias=False)
1975
+ )
1976
+ (input_projetor): Linear(in_features=1024, out_features=896, bias=True)
1977
+ )