程序员阿江(Relakkes)
ff8c92daad
chore: add copyright to every file
2025-11-18 12:24:02 +08:00
程序员阿江(Relakkes)
e89a6d5781
feat: cdp browser cleanup after crawler done
2025-11-17 12:21:53 +08:00
程序员阿江(Relakkes)
a1c5e07df8
fix: xhs sub comment bugfix #769
2025-11-17 11:47:33 +08:00
程序员阿江(Relakkes)
889fa01466
fix: bili词云图修复
2025-11-02 13:25:31 +08:00
persist-1
d3bebd039e
refactor(database): 调整数据库模块位置、调整初始化arg名称，并更新文档
...
- 将 db.py 数据库模块移入‘database/‘中，并修正所有调用代码
- 将初始化参数 `--init-db` 改为 `--init_db`
- 更新相关文档说明
2025-09-08 01:14:31 +08:00
persist-1
be306c6f54
refactor(database): 重构数据库存储实现，使用SQLAlchemy ORM替代原始SQL操作
...
- 删除旧的async_db.py和async_sqlite_db.py实现
- 新增SQLAlchemy ORM模型和数据库会话管理
- 统一各平台存储实现到_store_impl.py文件
- 添加数据库初始化功能支持
- 更新.gitignore和pyproject.toml依赖配置
- 优化文件存储路径和命名规范
2025-09-06 04:10:20 +08:00
程序员阿江(Relakkes)
a4d9aaa34a
refactor: xhs update
2025-07-21 21:26:16 +08:00
程序员阿江(Relakkes)
13b00f7a36
refactor: config update
2025-07-18 23:26:52 +08:00
gaoxiaobei
5daae04c7d
fix: conflict resolution errors
2025-07-17 16:54:17 +08:00
gaoxiaobei
6ced357096
Merge branch 'main' into dev
2025-07-17 06:45:30 +08:00
买定不离手
f5fbbb36ba
fix: 修复SQLite数据库初始化和关闭逻辑\n\n- 更新 main.py: 修复数据库初始化条件，支持sqlite选项\n- 更新 db.py: 添加SQLite数据库初始化和关闭支持
2025-07-14 03:36:48 +08:00
gaoxiaobei
e91ec750bb
feat: Enhance Bilibili crawler with retry logic and robustness
...
This commit introduces several improvements to enhance the stability and functionality of the Bilibili crawler.
- **Add Retry Logic:** Implement a retry mechanism with exponential backoff when fetching video comments. This makes the crawler more resilient to transient network issues or API errors.
- **Improve Error Handling:** Add a `try...except` block to handle potential `JSONDecodeError` in the Bilibili client, preventing crashes when the API returns an invalid response.
- **Ensure Clean Shutdown:** Refactor `main.py` to use a `try...finally` block, guaranteeing that the crawler and database connections are properly closed on exit, error, or `KeyboardInterrupt`.
- **Update Default Config:** Adjust default configuration values to increase concurrency, enable word cloud generation by default, and refine the Bilibili search mode for more practical usage.
2025-07-13 10:42:15 +08:00
程序员阿江(Relakkes)
e83b2422d9
feat: 支持playwright通过cdp协议连接本地chrome浏览器
...
docs: 增加uv来管理python依赖的文档
2025-06-25 23:22:39 +08:00
Relakkes
9fe3e47b0f
chore: 增加代码学习声明，严格禁止非法、禁止商业、不当用途
2024-10-20 00:43:25 +08:00
Relakkes
b7e57da0d2
feat: 知乎支持（关键词、评论）
2024-09-08 00:00:04 +08:00
Relakkes
d6fb255bdf
fix: xhs note detail error
2024-09-02 21:45:12 +08:00
Relakkes
a87094f2fd
feat: 百度贴吧架子 & 登录done
2024-08-05 18:51:51 +08:00
nelzomal
eace7d1750
improve base config reading command line arg logic
2024-06-09 18:51:36 +08:00
Relakkes
e367129b5f
fix: 修复pr#229的小bug
2024-04-13 13:48:29 +08:00
Tianci-King
1115b0d90c
feat(core): 新增控制爬虫 参数起始页面的页数start_page;perf(argparse): 向命令行解析器添加程序参数起始页面页数和关键字
2024-04-12 00:52:47 +08:00
jayeeliu@gmail.com
61ba8c5cc7
feat: 小红书支持通过博主ID采集笔记和评论，小红书type=search时支持配置按哪种排序方式获取笔记数据，小红书笔记增加视频地址和标签字段
2024-03-02 01:49:42 +08:00
ccgo
c09f9fef68
fix: 增加db.close()，解决抓取命令执行完不退出的问题
2024-02-22 00:11:41 +08:00
Relakkes
894dabcf63
refactor: 数据存储重构，分离不同类型的存储实现
2024-01-14 22:06:31 +08:00
Relakkes
eee81622ac
feat: 微博支持评论 & 指定帖子
2023-12-25 00:02:11 +08:00
Relakkes
c5b64fdbf5
feat: 微博爬虫帖子搜索完成
2023-12-24 17:57:48 +08:00
Relakkes
1cec23f73d
feat: 代理IP功能 Done
2023-12-08 00:10:04 +08:00
Relakkes
5aeee93fc5
feat: B站爬虫签名实现
2023-12-03 00:30:10 +08:00
Relakkes
986179b9c9
feat: 增加 IP 代理的最新实现
2023-12-02 16:14:36 +08:00
Relakkes
f08b2ceb76
feat: 1、命令行支持快手 2、快手playwright 代码 done
2023-11-24 00:04:33 +08:00
Relakkes
81bc8b51e2
feat: 抖音支持指定视频列表爬去
2023-11-18 22:07:30 +08:00
Relakkes
700946b28a
feat: 小红书增加指定帖子爬取功能
...
fix: 修复程序一些异常 bug
refactor: 优化部分代码逻辑
2023-11-18 13:38:11 +08:00
Relakkes
4ff2cf8661
refactor: 优化代码
2023-07-29 15:35:40 +08:00
Relakkes
e75707443b
feat: 增加配置项支持自由选择数据是否保存到关系型数据库中
2023-07-24 20:59:43 +08:00
Relakkes
2398a17e21
refactor: 优化抖音Crawler部分代码
...
fix: 日志初始化错误修复
2023-07-15 21:30:12 +08:00
Relakkes
b8093a2c0f
refactor:优化部分代码
...
feat: 增加IP代理账号池
2023-06-27 23:38:30 +08:00
Relakkes
1085a2a769
fix: 增加小红书登录两种形态下弹窗的兼容代码
2023-06-22 22:43:26 +08:00
ubuntu
79a0296312
handby&catch exception
2023-06-17 15:14:58 +08:00
Relakkes
8206f83639
feat: 小红书增加手机号自动登录模式
2023-06-16 20:10:29 +08:00
NanmiCoder
e82dcae02f
feat: 小红书笔记搜索，评论获取done
...
docs: update docs
Create .gitattributes
Update README.md
2023-06-12 20:37:24 +08:00
