injectlib/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2025-11-25 11:29:27 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update core.py，为爬取类型为detail和creator的任务，添加了和search任务一样的，用于转存up主信息的bilibili_store.update_up_info的函数调用

Browse Source 
正如`search`函数中一样，在调用`get_video_info_task`后，`bilibili_video`和`bilibili_up_info`信息都将获得。
原先的`get_specified_videos`在`detail`任务中仅保存了指定`bilibili_video`的信息，而`bilibili_up_info`信息尚未保存，`creator`任务的`get_creator_videos`中也调用了`get_specified_videos`获取指定创作者下所有的视频信息，同理也未保存`bilibili_up_info`信息。
所以只需为`get_specified_videos`添加一句`await bilibili_store.update_up_info(video_detail)`即可和`search`任务下获得的数据文件个数保持一致，不会缺少对应up主的个人信息。
已测试：
- 原先仅`search`任务下产生`*_creator.csv`、`*_contents.csv`、`*_comments.csv`，而`detail`和`creator`任务下缺少`*_creator.csv`文件。
- 此次提交后将使三种模式下的数据文件个数一致。
This commit is contained in:
翟持江
2025-01-19 19:55:18 +08:00
committed by GitHub
parent 2d93ec5a82
commit 0364b23b5b
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
media_platform/bilibili/core.py
View File

@@ -297,6 +297,7 @@ class BilibiliCrawler(AbstractCrawler):
if video_aid:
video_aids_list.append(video_aid)
await bilibili_store.update_bilibili_video(video_detail)
await bilibili_store.update_up_info(video_detail)
await self.get_bilibili_video(video_detail, semaphore)
await self.batch_get_video_comments(video_aids_list)