gaoxiaobei
d0d7293926
feat(bilibili): Add flexible search modes and fix limit logic
...
Refactors the Bilibili keyword search functionality to provide more flexible crawling strategies and corrects a flaw in how crawl limits were applied.
Previously, the `ALL_DAY` boolean flag offered a rigid choice for time-based searching and contained a logical issue where `CRAWLER_MAX_NOTES_COUNT` was incorrectly applied on a per-day basis instead of as an overall total.
This commit introduces the `BILI_SEARCH_MODE` configuration option with three distinct modes:
- `normal`: The default search behavior without time constraints.
- `all_in_time_range`: Maximizes data collection within a specified date range, replicating the original intent of `ALL_DAY=True`.
- `daily_limit_in_time_range`: A new mode that strictly enforces both the daily `MAX_NOTES_PER_DAY` and the total `CRAWLER_MAX_NOTES_COUNT` limits across the entire date range.
This change resolves the limit logic bug and gives users more precise control over the crawling process.
Changes include:
- Modified `config/base_config.py` to replace `ALL_DAY` with `BILI_SEARCH_MODE`.
- Refactored `media_platform/bilibili/core.py` to implement the new search mode logic.
2025-07-13 06:07:13 +08:00
gaoxiaobei
cad9fc7af8
feat: Add daily limit for video/post crawling in Bilibili and base config
2025-07-12 14:50:59 +08:00
Lei Cao
355ed183dd
增加选择微博搜索类型的配置
2025-07-05 22:14:31 +00:00
程序员阿江(Relakkes)
848df2b491
feat: other platfrom support the cdp mode
2025-07-03 17:13:32 +08:00
程序员阿江(Relakkes)
e83b2422d9
feat: 支持playwright通过cdp协议连接本地chrome浏览器
...
docs: 增加uv来管理python依赖的文档
2025-06-25 23:22:39 +08:00
Bowenwin
66843f216a
finish_all_for_expand_bili
2025-05-22 22:26:30 +08:00
Bowenwin
59619fff0a
finish_all
2025-05-22 22:06:06 +08:00
Bowenwin
44e3d370ff
fix_words
2025-05-22 20:31:48 +08:00
Bowenwin
a356358c21
get_fans_and_get_followings
2025-05-19 19:57:36 +08:00
Relakkes
b43d6b7b91
chore: update config
2025-02-12 10:58:48 +08:00
Relakkes
66a7ab1db8
refactor: bibi default to get without time data
2025-02-12 10:58:15 +08:00
翟持江
bf87821d4b
为 base_config.py 添加了是否开启按每一天进行爬取的选项
...
`bilibili 关键词搜索`仅返回 1000 条视频记录，共计 34 页，前 33 页视频记录数为 30，第 34 页视频记录数为 30。若要获取更多视频，需要设置更多筛选中的时间段选项，最高支持细分为 1 天的视频记录
---
此处添加了`START_DAY`与`END_DAY`以及`ALL_DAY`选项，在`ALL_DAY`为False时，使用原先关键词搜索策略，与原先版本保持不变；`ALL_DAY`为True时，从`START_DAY`与`END_DAY`中解析`client.py`中的search_video_by_keyword函数接收的`pubtime_begin_s`和`pubtime_end_s`参数，以实现最高支持细分为 1 天的视频记录，具体更改见之后提交的`client.py`和`core.py`
2025-01-15 18:06:16 +08:00
Relakkes
ea5223c708
feat: 知乎支持详情模式
2024-12-26 17:36:33 +08:00
liudongkai
33e7ef016d
feat: xhs 非代理模式下增加随机等待间隔, db存储模式下增加存储xsec_token字段
2024-12-05 21:10:31 +08:00
Relakkes
43dffeb2d1
feat: xhs帖子详情获取优化
2024-11-26 13:37:53 +08:00
Relakkes
6a96c00b4b
chore: config update & xinqqiu img update
2024-10-24 15:35:34 +08:00
unknown
7e53c4acfc
All_platform_comments_restrict
2024-10-23 16:32:02 +08:00
unknown
19269c66fd
xiaohongshu_comment_number_restrict
2024-10-22 20:33:10 +08:00
Relakkes
03e393949a
fix: xhs帖子详情问题更新
2024-10-20 00:59:08 +08:00
Relakkes
9fe3e47b0f
chore: 增加代码学习声明，严格禁止非法、禁止商业、不当用途
2024-10-20 00:43:25 +08:00
Relakkes
aa0f920369
feat: B站搜索接口增加发布日期筛选
2024-10-17 15:11:25 +08:00
Relakkes
da8f1c62b8
feat: 知乎支持创作者主页数据爬取(回答、文章、视频)
2024-10-16 21:02:27 +08:00
Relakkes
b7e57da0d2
feat: 知乎支持（关键词、评论）
2024-09-08 00:00:04 +08:00
Relakkes Yang
acb29add28
feat: 百度贴吧支持创作者主页帖子爬取
2024-08-24 11:03:23 +08:00
Relakkes
8adb593ba6
temp commit
2024-08-24 09:12:03 +08:00
Relakkes
ab7d8142af
feat: weibo支持指定创作者主页
2024-08-24 05:52:11 +08:00
Relakkes
f371675d47
fix: xhs指定笔记ID获取方式增加解析html方式，原来的由于xsec_token导致失效
2024-08-11 22:37:10 +08:00
Relakkes
3f42368c02
feat: 百度贴吧done
2024-08-08 14:19:32 +08:00
Relakkes
df0f5c1113
feat: 百度贴吧子评论done
2024-08-07 04:13:15 +08:00
Relakkes
026d81e131
feat: 百度贴吧一级评论done
2024-08-07 02:34:56 +08:00
Relakkes
1b585cb215
temp commit
2024-08-06 19:21:34 +08:00
Relakkes
a87094f2fd
feat: 百度贴吧架子 & 登录done
2024-08-05 18:51:51 +08:00
Relakkes
573ca9a659
feat: xhs笔记详情更新
2024-07-25 00:44:46 +08:00
Relakkes
548271e537
fix: 修复抖音中文搜索关键二次编码问题
2024-07-16 01:33:58 +08:00
helloteemo
6545a15ff3
feature: 支持小红书图片、视频下载
2024-07-11 22:56:30 +08:00
3DRX
c3a313563a
Update comment
2024-07-11 17:53:04 +08:00
522109452
6080c22a3d
feat: base_config 增加抖音发布时间配置
...
fix: 抖音排序类型枚举值
fix: 抖音offset计算问题
2024-06-14 14:13:39 +08:00
HIRO
fd7407cc29
Merge branch 'kuaishou'
2024-06-13 14:54:01 +08:00
HIRO
a001556ba7
快手指定创作者主页和二级评论
2024-06-13 14:49:07 +08:00
程序员阿江-Relakkes
131e68334d
Merge branch 'main' into main
2024-06-12 21:53:41 +08:00
程序员阿江-Relakkes
551e00c093
Merge pull request #295 from nelzomal/bili_creator
...
feat: support bilibili creator
2024-06-12 21:44:59 +08:00
nelzomal
111e08602c
feat: support bilibili creator
2024-06-12 16:48:19 +08:00
Rosyrain
7048f040c9
完成词云图生成函数并添加至存储逻辑中
2024-06-12 15:33:39 +08:00
nelzomal
985ea93caf
add few arg cmds
2024-06-12 10:53:03 +08:00
Relakkes
3c7c678d7a
feat: 抖音db存储增加parent_comment_id
2024-06-07 13:32:58 +08:00
Relakkes
478db4cc4b
feat: 抖音指定创作者done
2024-05-28 01:07:19 +08:00
Relakkes
764bafc626
feat: 抖音登录态检测逻辑更新支持
2024-05-23 22:15:14 +08:00
Relakkes
704ee8cf9d
docs: update README.md
2024-05-12 23:28:50 +08:00
Relakkes
d0efd4e4f0
feat: 快代理实现
2024-04-17 23:12:41 +08:00
程序员阿江-Relakkes
a341dc2aff
Merge pull request #229 from Tianci-King/main
...
feat(core): 新增控制爬虫参数起始页面的页数start_page;perf(argparse): 向命令行解析器添加程序参数…
2024-04-13 13:37:35 +08:00
