Commit Graph

44 Commits

Author SHA1 Message Date
j2rong4cn
6de15b6310 feat(stream): enhance GetRangeReaderFromLink rate limiting (#1528)
* feat(stream): enhance GetRangeReaderFromLink rate limiting

* refactor(stream): update GetRangeReaderFromMFile to return *model.FileRangeReader

* refactor(stream): simplify context error handling in RateLimitReader, RateLimitWriter, and RateLimitFile

* refactor(net): replace custom LimitedReadCloser with readers.NewLimitedReadCloser

* fix(model): update Link.ContentLength JSON tag for correct serialization

* docs(model): add clarification to FileRangeReader usage comment
2025-11-04 23:56:09 +08:00
ILoveScratch
febbcd6027 feat(cache): improve cache management (#1339)
* feat(cache): improve cache management

* feat(disk-usage): add cache

* feat(disk-usage): add refresh

* fix(disk-usage): cache with ttl

* feat(cache): implement KeyedCache and TypedCache for improved caching mechanism

* fix(copy): update object retrieval to use Get instead of GetUnwrap

* refactor(cache): simplify DirectoryCache structure and improve object management

* fix(cache): correct cache entry initialization and key deletion logic in TypedCache

* refactor(driver): remove GetObjInfo interface and simplify Link function logic
https://github.com/OpenListTeam/OpenList/pull/888/files#r2430925783

* fix(link): optimize link retrieval and caching logic

* refactor(cache): consolidate cache management and improve directory cache handling

* fix(cache): add cache control based on storage configuration in List function

* .

* refactor: replace fmt.Sprintf with strconv for integer conversions

* refactor(cache): enhance cache entry management with Expirable interface

* fix(cache): improve link reference acquisition logic to handle expiration

* refactor: replace OnlyLinkMFile with NoLinkSF in driver configurations and logic

* refactor(link): enhance link caching logic with dynamic type keys based on IP and User-Agent

* feat(drivers): add LinkCacheType to driver configurations for enhanced caching

* refactor(cache): streamline directory object management in cache operations

* refactor(cache): remove unnecessary 'dirty' field from CacheEntry structure

* refactor(cache): replace 'dirty' field with bitwise flags

* refactor(io): 调高SyncClosers.AcquireReference的优先级

* refactor(link): 优化链接获取逻辑,增加重

* refactor(link): 添加RequireReference字段以增强链接管理

* refactor(link): 移除MFile字段,改用RangeReader

* refactor: 移除不必要的NoLinkSF字段

* refactor(cache): 修改目录缓存的脏标志定义和更新逻辑

* feat(cache): add expiration gc

---------

Co-authored-by: KirCute <951206789@qq.com>
Co-authored-by: KirCute <kircute@foxmail.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
2025-10-18 21:47:18 +08:00
j2rong4cn
317d190b77 fix(ftp): create a new connection for each download (#989) 2025-08-06 20:35:01 +08:00
j2rong4cn
9ac0484bc0 perf(ftp): improve concurrent Link response; fix alias/local driver issues (#974) 2025-08-06 13:32:37 +08:00
j2rong4cn
8cf15183a0 perf: optimize upload (#554)
* pref(115,123): optimize upload

* chore

* aliyun_open, google_drive

* fix bug

* chore

* cloudreve, cloudreve_v4, onedrive, onedrive_app

* chore(conf): add `max_buffer_limit` option

* 123pan multithread upload

* doubao

* google_drive

* chore

* chore

* chore: 计算分片数量的代码

* MaxBufferLimit自动挡

* MaxBufferLimit自动挡

* 189pc

* errorgroup添加Lifecycle

* 查缺补漏

* Conf.MaxBufferLimit单位为MB

* 。

---------

Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-05 21:42:54 +08:00
j2rong4cn
cc01b410a4 perf(link): optimize concurrent response (#641)
* fix(crypt): bug caused by link cache

* perf(crypt,mega,halalcloud,quark,uc): optimize concurrent response link

* chore: 删除无用代码

* ftp

* 修复bug;资源释放

* 添加SyncClosers

* local,sftp,smb

* 重构,优化,增强

* Update internal/stream/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

* chore

* chore

* 优化,修复bug

* .

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-12 17:57:54 +08:00
j2rong4cn
ffb6c2a180 refactor: optimize stream, link, and resource management (#486)
* refactor: optimize stream, link, and resource management

* Link.MFile改为io.ReadSeeker类型

* fix (crypt): read on closed response body

* chore

* chore

* chore
2025-07-03 10:39:34 +08:00
MadDogOwner
874dc292ae fix(gomod): go modules with tagged versions (#499)
fix: go modules with tagged versions
2025-07-01 09:54:50 +08:00
Kuingsmile
fdcc2f136e chore: change module name to OpenListTeam/OpenList (#2)
* Enable blank issue

* chore(README.md): update docs (temporally)

* Update FUNDING.yml

* chore: purge README.md

* chore: change module name to OpenListTeam/OpenList

* fix: fix link errors

* chore: remove v3 in module name

* fix: resolve some conficts

* fix: resolve conficts

* docs: update with latest file

---------

Co-authored-by: ShenLin <773933146@qq.com>
Co-authored-by: Hantong Chen <cxwdyx620@gmail.com>
Co-authored-by: joshua <i@joshua.su>
Co-authored-by: Hantong Chen <70561268+cxw620@users.noreply.github.com>
2025-06-12 22:02:46 +08:00
KirCute_ECT
3b71500f23 feat(traffic): support limit task worker count & file stream rate (#7948)
* feat: set task workers num & client stream rate limit

* feat: server stream rate limit

* upgrade xhofe/tache

* .
2025-02-16 12:22:11 +08:00
KirCute_ECT
779c293f04 fix(driver): implement canceling and updating progress for putting for some drivers (#7847)
* fix(driver): additionally implement canceling and updating progress for putting for some drivers

* refactor: add driver archive api into template

* fix(123): use built-in MD5 to avoid caching full

* .

* fix build failed
2025-02-01 17:29:55 +08:00
Toby Shi
fcf2683112 feat(ftp): custom encoding (#6528 close #1260) 2024-06-16 16:58:02 +08:00
Andy Hsu
f6ab1f7f61 perf(ftp): non use SIZE FTP command (close #5150) 2023-09-03 18:47:32 +08:00
Andy Hsu
34fadecc2c fix(ftp): dead lock on Read (close #5128) 2023-08-31 15:10:47 +08:00
Sean
a3748af772 feat: misc improvements about upload/copy/hash (#5045)
general: add createTime/updateTime support in webdav and some drivers
general: add hash support in some drivers
general: cross-storage rapid-upload support
general: enhance upload to avoid local temp file if possible
general: replace readseekcloser with File interface to speed upstream operations
feat(aliyun_open): same as above
feat(crypt): add hack for 139cloud

Close #4934 
Close #4819 

baidu_netdisk needs to improve the upload code to support rapid-upload
2023-08-27 21:14:23 +08:00
Sean
3c21a9a520 feat: Crypt driver, improve http/webdav handling (#4884)
this PR has several enhancements, fixes, and features:
- [x] Crypt: a transparent encryption driver. Anyone can easily, and safely store encrypted data on the remote storage provider.  Consider your data is safely stored in the safe, and the storage provider can only see the safe, but not your data.
  - [x] Optional: compatible with [Rclone Crypt](https://rclone.org/crypt/). More ways to manipulate the encrypted data.
  - [x] directory and filename encryption
  - [x] server-side encryption mode (server encrypts & decrypts all data, all data flows thru the server)
- [x] obfuscate sensitive information internally
- [x] introduced a server memory-cached multi-thread downloader.
  - [x] Driver: **Quark** enabled this feature, faster load in any single thread scenario. e.g. media player directly playing from the link, now it's faster.
- [x] general improvement on HTTP/WebDAV stream processing & header handling & response handling
  - [x] Driver: **Mega** driver support ranged http header
  - [x] Driver: **Quark** fix bug of not closing HTTP request to Quark server while user end has closed connection to alist

## Crypt, a transparent Encrypt/Decrypt Driver. (Rclone Crypt compatible)

e.g.  
Crypt mount path ->  /vault 
Crypt remote path -> /ali/encrypted
Aliyun mount paht -> /ali

when the user uploads a.jpg to /vault, the data will be encrypted and saved to /ali/encrypted/xxxxx. And when the user wants to access a.jpg,  it's automatically decrypted, and the user can do anything with it.
Since it's Rclone Crypt compatible, users can download /ali/encrypted/xxxxx  and decrypt it with rclone crypt tool. Or the user can mount this folder using rclone, then mount the decrypted folder in Linux...

NB.  Some breaking changes is made to make it follow global standard, e.g. processing the HTTP header properly.

close #4679 
close #4827 

Co-authored-by: Sean He <866155+seanhe26@users.noreply.github.com>
Co-authored-by: Andy Hsu <i@nn.ci>
2023-08-02 14:40:36 +08:00
hcrgm
4d9a29bddd feat(ftp): support seek/range request (#3811) 2023-03-11 21:02:47 +08:00
Noah Hsu
105f22969c feat: support cancel for some drivers (close #2717) 2022-12-21 15:03:09 +08:00
foxxorcat
fb64f00640 refactor: obj name mapping and internal path processing (#2733)
* refactor:Prepare to remove the get interface

* feat:add obj Unwarp interface

* refactor:obj name mapping and program internal path processing

* chore: fix typo

* feat: unwrap get

* fix: no use op.Get to get parent id

* fix: set the path uniformly

Co-authored-by: Noah Hsu <i@nn.ci>
2022-12-17 19:49:05 +08:00
foxxorcat
33bae52fa1 refactor: optimize driver initialization need to manually deserialize and assign values, and remove redundant driver registration parameters (#2691)
* refactor: optimize driver initialization need to manually deserialize and assign values, and remove redundant driver registration parameters

* fix typo

Co-authored-by: Noah Hsu <i@nn.ci>
2022-12-13 18:03:30 +08:00
Noah Hsu
656f5f112c fix(ftp): nil pointer dereference (close #1722) 2022-09-20 22:23:22 +08:00
Noah Hsu
cd73e34ccc chore: optional other interface 2022-09-11 18:40:19 +08:00
Noah Hsu
a1c86b3350 chore!: change root folder 2022-09-04 13:22:42 +08:00
Noah Hsu
ffba5e0aec feat: add sftp driver (close #1466) 2022-09-04 12:43:52 +08:00
Noah Hsu
849de88e68 feat: add ftp driver 2022-09-03 22:07:08 +08:00
Noah Hsu
b76060570e refactor: init v3 2022-06-06 16:28:37 +08:00
Xhofe
87e339850d fix(ftp): remove dir (#1082) 2022-05-13 17:38:22 +08:00
微凉
feb42f1f4b 🔥 delete proxy interface 2022-02-02 18:03:54 +08:00
微凉
cb2a3c2b42 🎨 change proxy interface 2022-02-01 14:28:21 +08:00
微凉
cb23edc1fe 🐛 fix #462 check connect while get ftp client 2022-01-31 11:13:29 +08:00
微凉
6fd05d7d72 🐛 fix connMap not init 2022-01-30 00:55:12 +08:00
微凉
f26ac57569 🐛 fix ftp conn not store 2022-01-30 00:04:31 +08:00
微凉
2473309a51 🎇 execute save while delete account 2022-01-15 19:36:37 +08:00
微凉
424ec10692 🐛 fix ftp download error 2022-01-13 22:56:07 +08:00
微凉
7cf30836bf 🎨 Optimize code structure 2022-01-03 20:06:36 +08:00
微凉
e7ba289d06 🐛 fix upload delete cache 2022-01-02 18:17:04 +08:00
微凉
939c9cd5ac 🚧 check upload file 2021-12-31 14:05:35 +08:00
微凉
d71ed4d775 🎇 support webdav driver 2021-12-30 21:39:17 +08:00
微凉
3331462229 local sort for no sort param driver 2021-12-30 16:15:57 +08:00
微凉
6a54ed87f3 🎨 change NeedSetLink to NoNeedSetLink 2021-12-29 19:47:47 +08:00
Xhofe
26b4766da7 remove . and .. for FTP 2021-12-25 19:22:05 +08:00
微凉
d00f75c814 🎨 change link interface 2021-12-19 20:00:53 +08:00
微凉
1c65588b4a 🎇 webdav update delete cache 2021-12-16 22:02:45 +08:00
微凉
d49f92b542 ftp support 2021-12-16 18:03:58 +08:00