add comment api and doc, fix CI

DEVELOPMENT.md

@@ -1,6 +1,6 @@
 # 项目手册
 
-# 部署运行
+# bot 部署运行
 
 ## docker-compose
 
@@ -55,8 +55,6 @@ pip install -r requirements.txt
 python3 web/prepare/convert_db.py
 ```
 
-**不再兼容旧版本数据**
-
 ### 4. 运行
 
 ```bash
@@ -67,25 +65,16 @@ python /path/to/YYeTsBot/yyetsbot/bot.py
 
 参考 `yyets.service`
 
-### 6. 网站部署运行方式
+# 网站部署运行方式
 
 参考 `worker`和`web`目录下的 `README`。需要注意，cf worker已经停止开发。
 
-
+# 添加新的资源网站
-## 添加新的资源网站
 
 欢迎各位开发提交新的资源网站！方法非常简单，重写 `BaseFansub`，实现`search_preview`和`search_result`，按照约定的格式返回数据。
 
 然后把类名字添加到 `FANSUB_ORDER` 就可以了！是不是很简单！
 
-## bot无响应
-
-有时不知为何遇到了bot卡死，无任何反馈。😂~~这个时候需要client api了~~😂
-
-原因找到了，是因为有时爬虫会花费比较长的时间，然后pytelegrambotapi默认只有两个线程，那么后续的操作就会被阻塞住。
-
-临时的解决办法是增加线程数量，长期的解决办法是使用celery分发任务。
-
 # 网站开发手册
 
 ## 接口列表
@@ -131,3 +120,63 @@ python /path/to/YYeTsBot/yyetsbot/bot.py
 * [网站实时数据，MongoDB](https://yyets.dmesg.app/data/yyets_mongo.gz)
 * [MySQL](https://yyets.dmesg.app/data/yyets_mysql.zip)
 * [SQLite](https://yyets.dmesg.app/data/yyets_sqlite.zip)
+
+
+# 评论API
+
+## 1. 获取评论
+GET `/api/comments`
+
+分页，支持URL参数：
+* resource_id: 资源id
+* size: 每页评论数量，默认5（或者其他数值）
+* page: 当前页
+
+返回
+```json
+{
+    "data": [
+        {
+            "date": "2018-09-18 11:12:15",
+            "username": "uuua2",
+            "content": "tdaadd",
+            "id": 2
+        },
+        {
+            "date": "2018-09-01 11:12:15",
+            "username": "abcd",
+            "content": "tdaadd",
+            "id": 1
+        }
+    ],
+    "count": 2,
+    "resource_id": 39301
+}
+
+```
+
+## 2. 获取验证码
+GET `/api/captcha?id=1234abc`，id是随机生成的字符串
+API 返回字符串，形如 `data:image/png;base64,iVBORw0KGgoAAA....`
+
+## 3. 提交评论
+POST `/api/comments`
+只有登录用户才可以发表评论，检查cookie `username` 是否为空来判断是否为登录用户；未登录用户提示“请登录后发表评论”
+
+body `resource_id` 从URL中获取，id是上一步验证码的那个id， `captcha` 是用户输入的验证码
+```json
+{
+    "resource_id": 39301,
+    "content": "评论内容",
+    "id": "1234abc",
+    "captcha": "38op"
+}
+```
+
+返回 HTTP 201添加评论成功，403/401遵循HTTP语义
+
+```json
+{
+    "message": "评论成功/评论失败/etc"
+}
+```

Dockerfile

@@ -1,8 +1,8 @@
 FROM python:3.9-alpine as builder
 
-RUN apk update && apk add  --no-cache tzdata ca-certificates alpine-sdk libressl-dev libffi-dev cargo
+RUN apk update && apk add  --no-cache tzdata ca-certificates alpine-sdk libressl-dev libffi-dev cargo && \
-# build cryptography separately because it's very slow
+    apk add tiff-dev jpeg-dev openjpeg-dev zlib-dev freetype-dev lcms2-dev \
-RUN pip3 install --user cryptography==3.4.7
+    libwebp-dev tcl-dev tk-dev harfbuzz-dev fribidi-dev libimagequant-dev libxcb-dev libpng-dev
 
 COPY requirements.txt /requirements.txt
 RUN pip3 install  --user -r /requirements.txt && rm /requirements.txt
@@ -15,7 +15,7 @@ COPY --from=builder /root/.local /usr/local
 COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
 COPY . /YYeTsBot
-RUN apk update && apk add --no-cache libressl
+RUN apk update && apk add --no-cache libressl jpeg-dev openjpeg-dev libimagequant-dev tiff-dev freetype-dev libxcb-dev
 
 ENV TZ=Asia/Shanghai
 WORKDIR /YYeTsBot/yyetsbot

README.md

@@ -4,18 +4,12 @@
 
 [![codecov](https://codecov.io/gh/tgbot-collection/YYeTsBot/branch/master/graph/badge.svg?token=ZL1GCIF95D)](https://codecov.io/gh/tgbot-collection/YYeTsBot)
 
-**⚠️⚠️无法访问？自2021年3月22日起，本站间歇性地被墙啦！⬇️**
-
-🧱 <img src="https://gfw.dmesg.app/get?url=https://yyets.dmesg.app" width = "16" height = "16" alt="gfw status"/>
-
-
 * 人人影视bot，[戳我使用](https://t.me/yyets_bot)
 
 * 人人影视分享站，[戳我使用](https://yyets.dmesg.app/)
 
 机器人和网站由我长期维护，如果遇到问题可以提issue。
 
-
 # 使用说明
 
 直接发送想要看的剧集名称就可以了，可选分享网页或者链接（ed2k和磁力链接）。
@@ -94,7 +88,7 @@ yyets_offline - 人人影视离线数据
 * 捐助我，[爱发电？](https://afdian.net/@BennyThink)
 
 # 感谢
-[Thanks](THANKS.md)
+感谢所有支持本项目的人！
 
 # License
 [MIT](LICENSE)

THANKS.md

@@ -1,78 +0,0 @@
-# Thannks
-
-Special thanks to these following supporters!
-
-* 爱发电用户_cm5C
-* gin
-* aska
-* TC
-* sanarry
-* alan444
-* Mount
-* 爱发电用户_3Mku
-* toupucai
-* Ztachi
-* 爱发电用户_mhRY
-* 企鹅仔
-* Charlie
-* wabaman
-* zhangjichi007
-* 爱发电用户_vRYW
-* 爱发电用户_R8jG
-* 爱发电用户_yMxm
-* 西二切
-* 360Tencent
-* 慕星
-* Saul
-* 除了感谢还能说啥
-* Reimu
-* 朗冥其
-* 放牛的牛腩
-* changsr
-* 走走
-* 爱发电用户_Ad9M
-* 浪迹天涯
-* Iceyr
-* Ryan
-* 晨和我想
-* tallulah-dong
-* samlemitchell
-* 爱发电用户_3tar
-* 爱发电用户_kgCv
-* 爱发电用户_YCQe
-* Moreli
-* 爱发电用户_REKk
-* 唐小胖
-* chandy
-* alyzq
-* jcwwww233
-* 爱发电用户_dWTg
-* iason4994
-* 榕树籽
-* lase_mary
-* Mia
-* 爱发电用户_Suq4
-
-以及所有点赞、赞助和分享的可爱的你们！
-
-# afdian
-
-```javascript
-
-let md = "\n"
-let nameList = document.getElementsByClassName("avatar-name black-8 gl-hover-text-purple flex-box flex-align-items-center text-overflow-1")
-for (let i = 1; i < nameList.length; i++) {
-    md += "* " + nameList[i].textContent.trim() + "\n"
-}
-copy(md)
-```
-# buy me a coffee
-```javascript
-let md = "\n"
-let nameList = document.getElementsByClassName("supp-name limit-text-line limit-text-line-1 mh-22 mg-b-0 av-heavy color-0D0 xs-av-roman")
-for (let i = 1; i < nameList.length; i++) {
-    md += "* " + nameList[i].textContent.trim() + "\n"
-}
-copy(md)
-
-```

requirements.txt

@@ -7,6 +7,7 @@ apscheduler==3.6.3
 pymongo==3.11.2
 tornado==6.0.4
 redis==3.5.3
+captcha==0.3
 
 passlib==1.7.4
 cryptography==3.4.7

tests/test_bot.py

@@ -14,10 +14,10 @@ from unittest import mock
 from telebot.types import Message
 
 sys.path.append("../yyetsbot")
-import yyetsbot as mybot
+import yyetsbot
 
 
-@mock.patch("bot.bot")
+@mock.patch("yyetsbot.bot")
 class TestStartHandler(unittest.TestCase):
     @classmethod
     def setUpClass(cls) -> None:
@@ -29,7 +29,7 @@ class TestStartHandler(unittest.TestCase):
         pass
 
     def test_start(self, b):
-        mybot.send_welcome(self.message)
+        yyetsbot.send_welcome(self.message)
         self.assertEqual(1, b.send_message.call_count)
         self.assertEqual(1, b.send_chat_action.call_count)
         self.assertEqual(self.message.chat.id, b.send_message.call_args.args[0])

tests/test_fansub.py

@@ -8,7 +8,7 @@ import requests_mock
 from unittest import mock
 
 sys.path.append("../yyetsbot")
-import bot as _
+import yyetsbot as _
 
 from fansub import BaseFansub, YYeTsOnline, YYeTsOffline
 

web/server.py

@@ -11,7 +11,12 @@ import os
 import contextlib
 import logging
 import json
+import re
 import time
+import string
+import random
+import base64
+
 from urllib import request
 from datetime import date, timedelta
 from http import HTTPStatus
@@ -26,6 +31,7 @@ from tornado import web, ioloop, httpserver, gen, options, escape
 from tornado.log import enable_pretty_logging
 from tornado.concurrent import run_on_executor
 from passlib.hash import pbkdf2_sha256
+from captcha.image import ImageCaptcha
 
 from crypto import decrypt
 
@@ -36,6 +42,7 @@ if os.getenv("debug"):
     logging.basicConfig(level=logging.DEBUG)
 
 escape.json_encode = lambda value: json.dumps(value, ensure_ascii=False)
+predefined_str = re.sub(r"[1l0oOI]", "", string.ascii_letters + string.digits)
 
 
 class Mongo:
@@ -58,6 +65,7 @@ class Redis:
 
 class BaseHandler(web.RequestHandler):
     mongo = Mongo()
+    redis = Redis()
 
     def write_error(self, status_code, **kwargs):
         if status_code in [HTTPStatus.FORBIDDEN,
@@ -175,7 +183,7 @@ class UserHandler(BaseHandler):
             try:
                 self.mongo.db["users"].insert_one(dict(username=username, password=hash_value,
                                                        date=time.asctime(),
-                                                       ip=self.request.headers.get("X-Real-IP"),
+                                                       ip=(AntiCrawler(self).get_real_ip()),
                                                        browser=self.request.headers['user-agent']
                                                        )
                                                   )
@@ -394,6 +402,118 @@ class NameHandler(BaseHandler):
         self.write(resp)
 
 
+class CommentHandler(BaseHandler):
+    executor = ThreadPoolExecutor(100)
+
+    @run_on_executor()
+    def get_comment(self):
+        resource_id = int(self.get_argument("resource_id", "0"))
+        size = int(self.get_argument("size", "5"))
+        page = int(self.get_argument("page", "1"))
+        if not resource_id:
+            self.set_status(HTTPStatus.BAD_REQUEST)
+            return {"status": False, "message": "请提供resource id"}
+        # page 1 size 5 - latest, latest-page*size+1
+        count = self.mongo.db["comment"].count_documents({"resource_id": resource_id})
+        if page == 1:
+            start = count
+        else:
+            start = count - (page - 1) * size
+        filter_range = list(range(start, count - page * size, -1))
+        data = self.mongo.db["comment"].find(
+            {"resource_id": resource_id, "id": {"$in": filter_range}},
+            projection={"_id": False, "ip": False, "browser": False}
+        ).sort("id", pymongo.DESCENDING)
+
+        return {
+            "data": list(data),
+            "count": count,
+            "resource_id": resource_id
+        }
+
+    @run_on_executor()
+    def add_comment(self):
+        payload = json.loads(self.request.body)
+        captcha = payload["captcha"]
+        captcha_id = payload["id"]
+        content = payload["content"]
+        resource_id = payload["resource_id"]
+
+        result = CaptchaHandler.verify_code(captcha_id, captcha)
+        real_ip = AntiCrawler(self).get_real_ip()
+        username = self.get_secure_cookie("username")
+        exists = self.mongo.db["yyets"].find_one({"data.info.id": resource_id})
+
+        if not result["status"]:
+            self.set_status(HTTPStatus.BAD_REQUEST)
+            return result
+        if not exists:
+            self.set_status(HTTPStatus.NOT_FOUND)
+            return {"status": False, "message": "资源不存在"}
+        if not username:
+            self.set_status(HTTPStatus.UNAUTHORIZED)
+            return {"status": False, "message": "请先登录再评论"}
+
+        # one comment one document
+        newest = self.mongo.db["comment"].find({"resource_id": resource_id}).sort("id", pymongo.DESCENDING)
+        newest = list(newest)
+        new_id = newest[0]["id"] + 1 if newest else 1
+
+        construct = {
+            "username": username.decode("u8"),
+            "ip": real_ip,
+            "date": time.asctime(),
+            "browser": self.request.headers['user-agent'],
+            "content": content,
+            "id": new_id,
+            "resource_id": resource_id
+        }
+        self.mongo.db["comment"].insert_one(construct)
+        self.set_status(HTTPStatus.CREATED)
+        return {"message": "评论成功"}
+
+    @gen.coroutine
+    def get(self):
+        resp = yield self.get_comment()
+        self.write(resp)
+
+    @gen.coroutine
+    def post(self):
+        resp = yield self.add_comment()
+        self.write(resp)
+
+
+class CaptchaHandler(BaseHandler):
+    executor = ThreadPoolExecutor(10)
+
+    @run_on_executor()
+    def get_captcha(self):
+        request_id = self.get_argument("id", None)
+        if request_id is None:
+            self.set_status(HTTPStatus.BAD_REQUEST)
+            return "Please supply id parameter."
+        chars = "".join([random.choice(predefined_str) for _ in range(4)])
+        image = ImageCaptcha()
+        data = image.generate(chars)
+        self.redis.r.set(request_id, chars, ex=60 * 10)
+        return f"data:image/png;base64,{base64.b64encode(data.getvalue()).decode('ascii')}"
+
+    @gen.coroutine
+    def get(self):
+        resp = yield self.get_captcha()
+        self.write(resp)
+
+    @classmethod
+    def verify_code(cls, request_id, user_input):
+        correct_code = cls.redis.r.get(request_id)
+        if not correct_code:
+            return {"status": False, "message": "验证码已过期"}
+        if user_input == correct_code:
+            return {"status": True, "message": "验证通过"}
+        else:
+            return {"status": False, "message": "验证码错误"}
+
+
 class MetricsHandler(BaseHandler):
     executor = ThreadPoolExecutor(100)
 
@@ -551,6 +671,8 @@ class RunServer:
         (r'/api/top', TopHandler),
         (r'/api/user', UserHandler),
         (r'/api/name', NameHandler),
+        (r'/api/comment', CommentHandler),
+        (r'/api/captcha', CaptchaHandler),
         (r'/api/metrics', MetricsHandler),
         (r'/api/grafana/', GrafanaIndexHandler),
         (r'/api/grafana/search', GrafanaSearchHandler),

web/test.py

@@ -0,0 +1,26 @@
+#!/usr/local/bin/python3
+# coding: utf-8
+
+# YYeTsBot - test.py
+# 5/31/21 13:32
+#
+
+__author__ = "Benny <benny.think@gmail.com>"
+
+import pymongo
+
+client = pymongo.MongoClient()
+db = client["zimuzu"]
+col = db["comment"]
+
+for i in range(1, 18):
+    data = {
+        "username": "Benny",
+        "ip": "127.0.0.1",
+        "date": "Mon May 31 16:58:21 2021",
+        "browser": "PostmanRuntime/7.28.0",
+        "content": f"评论{i}",
+        "id": i,
+        "resource_id": 10004
+    }
+    col.insert_one(data)