injectlib/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2025-11-25 11:29:27 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #229 from Tianci-King/main

Browse Source 
feat(core): 新增控制爬虫参数起始页面的页数start_page;perf(argparse): 向命令行解析器添加程序参数…
This commit is contained in:
程序员阿江-Relakkes
2024-04-13 13:37:35 +08:00
committed by GitHub
parent da949f0c7a 1115b0d90c
commit a341dc2aff
9 changed files with 94 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
config/base_config.py
View File

@@ -27,6 +27,9 @@ SAVE_DATA_OPTION = "json" # csv or db or json
# 用户浏览器缓存的浏览器文件配置
USER_DATA_DIR = "%s_user_data_dir" # %s will be replaced by platform name
# 爬取开始页数 默认从第一页开始
START_PAGE = 1
# 爬取视频/帖子的数量控制
CRAWLER_MAX_NOTES_COUNT = 20