injectlib/OpenList
1
0
Fork 0
mirror of https://github.com/OpenListTeam/OpenList.git synced 2025-11-25 03:15:19 +08:00
Code Issues Packages Projects Releases Wiki Activity
2,602 Commits 18 Branches 147 Tags
main
Commit Graph

39 Commits

Author SHA1 Message Date
KirCute
ee2c77acd8 fix(archive/zip): user specific encoding for non-EFS zips (#1599) 
* fix(archive/zip): user specific encoding for non-EFS zips

* fix(stream): simplify head cache initialization and improve reader retrieval logic

* fix: support multipart zips (.z01)

* chore(deps): update github.com/KirCute/zip to v1.0.1

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
 2025-11-10 19:08:50 +08:00
j2rong4cn
a1f1f98f94 refactor(stream): simplify code (#1590) 
* refactor(stream): simplify Close method and update SeekableStream to use RangeReader interface

* refactor(stream):  improve RangeRead comments for clarity
 2025-11-06 20:06:48 +08:00
j2rong4cn
174eae802a perf(stream): optimize CacheFullAndWriter for better memory management (#1584) 
* perf(stream): optimize CacheFullAndWriter for better memory management

* fix(stream): ensure proper seek handling in CacheFullAndWriter for improved data integrity
 2025-11-05 12:16:09 +08:00
j2rong4cn
6de15b6310 feat(stream): enhance GetRangeReaderFromLink rate limiting (#1528) 
* feat(stream): enhance GetRangeReaderFromLink rate limiting

* refactor(stream): update GetRangeReaderFromMFile to return *model.FileRangeReader

* refactor(stream): simplify context error handling in RateLimitReader, RateLimitWriter, and RateLimitFile

* refactor(net): replace custom LimitedReadCloser with readers.NewLimitedReadCloser

* fix(model): update Link.ContentLength JSON tag for correct serialization

* docs(model): add clarification to FileRangeReader usage comment
 2025-11-04 23:56:09 +08:00
ILoveScratch
febbcd6027 feat(cache): improve cache management (#1339) 
* feat(cache): improve cache management

* feat(disk-usage): add cache

* feat(disk-usage): add refresh

* fix(disk-usage): cache with ttl

* feat(cache): implement KeyedCache and TypedCache for improved caching mechanism

* fix(copy): update object retrieval to use Get instead of GetUnwrap

* refactor(cache): simplify DirectoryCache structure and improve object management

* fix(cache): correct cache entry initialization and key deletion logic in TypedCache

* refactor(driver): remove GetObjInfo interface and simplify Link function logic
https://github.com/OpenListTeam/OpenList/pull/888/files#r2430925783

* fix(link): optimize link retrieval and caching logic

* refactor(cache): consolidate cache management and improve directory cache handling

* fix(cache): add cache control based on storage configuration in List function

* .

* refactor: replace fmt.Sprintf with strconv for integer conversions

* refactor(cache): enhance cache entry management with Expirable interface

* fix(cache): improve link reference acquisition logic to handle expiration

* refactor: replace OnlyLinkMFile with NoLinkSF in driver configurations and logic

* refactor(link): enhance link caching logic with dynamic type keys based on IP and User-Agent

* feat(drivers): add LinkCacheType to driver configurations for enhanced caching

* refactor(cache): streamline directory object management in cache operations

* refactor(cache): remove unnecessary 'dirty' field from CacheEntry structure

* refactor(cache): replace 'dirty' field with bitwise flags

* refactor(io): 调高SyncClosers.AcquireReference的优先级

* refactor(link): 优化链接获取逻辑，增加重

* refactor(link): 添加RequireReference字段以增强链接管理

* refactor(link): 移除MFile字段，改用RangeReader

* refactor: 移除不必要的NoLinkSF字段

* refactor(cache): 修改目录缓存的脏标志定义和更新逻辑

* feat(cache): add expiration gc

---------

Co-authored-by: KirCute <951206789@qq.com>
Co-authored-by: KirCute <kircute@foxmail.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
 2025-10-18 21:47:18 +08:00
j2rong4cn
2edc446ced feat(stream): support using temporary files as large buffer (#1399) 
feat(stream): refactor StreamSectionReader to support using temporary files as large buffer
 2025-10-01 18:43:20 +08:00
ILoveScratch
189cebe4c9 feat(drivers): add MediaFire driver support (#1322) 
* feat(drivers): add MediaFire driver support (#9319)

- Implement complete MediaFire storage driver
- Add authentication via session_token and cookie
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload API with multi-unit transfers
- Add proper error handling and progress reporting

Co-authored-by: Da3zKi7 <da3zki7@duck.com>

* fix(mediafire): fix code errors in mediafire

* fix(mediafire): fix code errors in mediafire

* fix(drivers): add session renewal cron for MediaFire driver (#9321)

- Implement automatic session token renewal every 6-9 minutes
- Add validation for required SessionToken and Cookie fields in Init
- Handle session expiration by calling renewToken on validation failure
- Prevent storage failures due to MediaFire session timeouts

Fixes session closure issues that occur after server restarts or extended periods.

Co-authored-by: Da3zKi7 <da3zki7@duck.com>

* docs: restore README changes

Signed-off-by: ILoveScratch <ilovescratch@foxmail.com>

* fix

* fix

* fix: add stream upload limit

* fix

* fix: clear action token on drop and refactor header setting

* feat(drivers/mediafire): optimize file caching - support direct stream processing

- Remove forced caching to *os.File type
- Support generic model.File interface for better flexibility
- Improve upload efficiency by avoiding unnecessary file conversions
- Fix return type to use model.Object instead of model.ObjThumb

* feat(drivers/mediafire): improve global rate limiting

- Ensure all API methods properly use context for rate limiting
- Fix context parameter usage in getDirectDownloadLink, getActionToken, getFileByHash
- Maintain consistent rate limiting across all MediaFire API calls

* feat(drivers/mediafire): unify return types - remove unnecessary ObjThumb

- Change MakeDir, Rename, Copy methods to return model.Object instead of model.ObjThumb
- Remove empty Thumbnail fields where not meaningful
- Keep ObjThumb only for fileToObj (List operations) which provides actual thumbnail URLs
- Improve code consistency and reduce unnecessary wrapper objects

* refactor(drivers/mediafire): extract common error handling logic

- Add checkAPIResult helper function to reduce code duplication
- Replace repetitive MediaFire API error checks with centralized function
- Maintain specific error messages for unique cases (token, upload, search)
- Improve code maintainability and consistency

* enhance(drivers/mediafire): improve quick upload implementation

- Add null check for existingFile to prevent potential issues
- Improve error handling in quick upload - continue normal upload if search fails
- Add detailed comments explaining quick upload logic
- Optimize getExistingFileInfo with clearer fallback strategy
- Ensure upload reliability even when file search encounters issues

* refactor(drivers/mediafire): optimize request method reusability

- Extract common HTTP request logic into apiRequest method
- Reduce code duplication between getForm and postForm methods
- Maintain backward compatibility with existing method signatures
- Centralize rate limiting and header management
- Support extensible HTTP method handling

* docs(drivers/mediafire): add comprehensive English comments

- Add function-level comments for all major driver methods
- Document Init, List, Link, MakeDir, Move, Rename, Copy, Remove, Put methods
- Add comments for key utility functions including session token management
- Improve code readability and maintainability for community collaboration
- Follow Go documentation conventions with clear, concise descriptions

* perf(mediafire): optimize memory allocation and type assertion performance

- Pre-allocate slice capacity in getFiles and bitmap conversion to reduce reallocations
- Cache file type check in uploadUnits to avoid repeated type assertions
- Add uploadSingleUnitOptimized for os.File to eliminate redundant type checks
- Optimize string to int conversion with proper error handling
- Improve memory efficiency in file upload operations

* fix(mediafire): upload without cache

* feat(mediafire): add rate limiting to all API methods

- Add WaitLimit(ctx) calls to all driver methods: List, Link, MakeDir, Move, Rename, Copy, Remove, Put
- Ensure consistent rate limiting across all MediaFire API interactions
- Follow project standard pattern used by other drivers

* feat(mediafire): improve error handling consistency

- Add context parameter to all HTTP API functions for proper context propagation
- Update getForm, postForm and apiRequest to accept context parameter
- Fix rate limiting to use caller context instead of background context
- Ensure consistent error handling patterns across all API calls
- Improve cancellation and timeout support

* feat(mediafire): refactor resumableUpload to use io.ReadSeeker and improve upload handling

* fix(mediafire): release section reader

* feat: add disk usage

* feat(drivers/mediafire): support concurrent upload (#1387)

* feat(drivers): add MediaFire driver with concurrent upload support

- Implement complete MediaFire storage driver with session token authentication
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload with intelligent and multi-unit transfers
- Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
- Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
- Include rate limiting and retry logic for API requests
- Add proper error handling and progress reporting
- Handle MediaFire's bitmap-based resumable upload protocol

Closes PR #1322

* feat(stream): add DiscardSection method to StreamSectionReader for skipping data

* feat(mediafire): refactor resumableUpload logic for improved upload handling and error management

* fix(mediafire): stop cron job and clear action token in Drop method

* .

* fix(mediafire): optimize buffer sizing logic in uploadUnits method

* fix(docs): remove duplicate MediaFire

* fix(mediafire): revert 'optimization', large files should not be fully chached.

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Da3zKi7 <da3zki7@duck.com>
Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

* fix(mediafire): optimize hash calculation in uploadUnits function

* feat(drivers/mediafire): support concurrent upload  (#1366)

* feat(drivers): add MediaFire driver with concurrent upload support

- Implement complete MediaFire storage driver with session token authentication
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload with intelligent and multi-unit transfers
- Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
- Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
- Include rate limiting and retry logic for API requests
- Add proper error handling and progress reporting
- Handle MediaFire's bitmap-based resumable upload protocol

Closes PR #1322

* feat(stream): add DiscardSection method to StreamSectionReader for skipping data

* feat(mediafire): refactor resumableUpload logic for improved upload handling and error management

* fix(mediafire): stop cron job and clear action token in Drop method

* .

* fix(mediafire): optimize buffer sizing logic in uploadUnits method

* fix(docs): remove duplicate MediaFire

* fix(mediafire): revert 'optimization', large files should not be fully chached.

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

---------

Signed-off-by: ILoveScratch <ilovescratch@foxmail.com>
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: Da3zKi7 <da3zki7@duck.com>
Co-authored-by: KirCute <951206789@qq.com>
Co-authored-by: Suyunmeng <Susus0175@proton.me>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
 2025-09-30 21:55:41 +08:00
KirCute
3936e736e6 feat(drivers): add a driver that divides large files into multiple chunks (#1153) 2025-09-19 19:27:35 +08:00
TwoOnefour
cbbb5ad231 fix(stream): http chucked upload issue (#1152) 
* fix(stream): http chucked upload issue

* fix(stream): use MmapThreshold

* fix(stream): improve caching mechanism and handle size=0 case

* fix bug

* fix(buffer): optimize ReadAt method for improved performance

* fix(upload): handle Content-Length and File-Size headers for better size management

* fix(189pc): 移除重复限速

* fix(upload): handle negative file size during streaming uploads

* fix(upload): update header key from File-Size to X-File-Size for size retrieval

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
 2025-09-15 19:36:16 +08:00
j2rong4cn
016ed90efa feat(stream): fast buffer freeing for large cache (#1053) 
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-16 17:19:52 +08:00
j2rong4cn
57fceabcf4 perf(stream): improve file stream range reading and caching mechanism (#1001) 
* perf(stream): improve file stream range reading and caching mechanism

* 。

* add bytes_test.go

* fix(stream): handle EOF and buffer reading more gracefully

* 注释

* refactor: update CacheFullAndWriter to accept pointer for UpdateProgress

* update tests

* Update drivers/google_drive/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

* 更优雅的克隆Link

* 修复stream已缓存但无法重复读取

* 将Bytes类型重命名为Reader

* 修复栈溢出

* update tests

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-11 23:41:22 +08:00
j2rong4cn
9ac0484bc0 perf(ftp): improve concurrent Link response; fix alias/local driver issues (#974) 2025-08-06 13:32:37 +08:00
j2rong4cn
8cf15183a0 perf: optimize upload (#554) 
* pref(115,123): optimize upload

* chore

* aliyun_open, google_drive

* fix bug

* chore

* cloudreve, cloudreve_v4, onedrive, onedrive_app

* chore(conf): add `max_buffer_limit` option

* 123pan multithread upload

* doubao

* google_drive

* chore

* chore

* chore: 计算分片数量的代码

* MaxBufferLimit自动挡

* MaxBufferLimit自动挡

* 189pc

* errorgroup添加Lifecycle

* 查缺补漏

* Conf.MaxBufferLimit单位为MB

* 。

---------

Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
 2025-08-05 21:42:54 +08:00
pnparadise
86324d2d6b fix(net): ensure accurate content-length in response (#749) 
* fix(fs): ensure accurate content-length in http2 requests

Chrome browsers were unable to preview thumbnails, reporting an
'ERR_HTTP_2_PROTOCOL_ERROR'. This was caused by an incorrect
content-length header in the server's response for thumbnail images.

This commit corrects the content-length calculation, allowing
Chrome and other compliant clients to render thumbnails correctly.

* fix(net): ensure accurate content-length in response

* 补缺

* .

---------

Co-authored-by: zhiqiang.huang <zhiqiang.tech@gmail.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
 2025-07-19 20:36:27 +08:00
j2rong4cn
0c461991f9 chore: standardize context keys with custom ContextKey type (#697) 
* chore: standardize context keys with custom ContextKey type

* fix bug

* 使用Request.Context
 2025-07-14 23:55:17 +08:00
j2rong4cn
cc01b410a4 perf(link): optimize concurrent response (#641) 
* fix(crypt): bug caused by link cache

* perf(crypt,mega,halalcloud,quark,uc): optimize concurrent response link

* chore: 删除无用代码

* ftp

* 修复bug；资源释放

* 添加SyncClosers

* local,sftp,smb

* 重构，优化，增强

* Update internal/stream/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

* chore

* chore

* 优化，修复bug

* .

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-07-12 17:57:54 +08:00
Seven
3838ef0663 feat(traffic): update progress when caching file (#646) 
* feat(traffic): update progress when caching file

* 调整参数位置和命名

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
 2025-07-08 21:41:45 +08:00
j2rong4cn
ffb6c2a180 refactor: optimize stream, link, and resource management (#486) 
* refactor: optimize stream, link, and resource management

* Link.MFile改为io.ReadSeeker类型

* fix (crypt): read on closed response body

* chore

* chore

* chore
 2025-07-03 10:39:34 +08:00
MadDogOwner
874dc292ae fix(gomod): go modules with tagged versions (#499) 
fix: go modules with tagged versions
 2025-07-01 09:54:50 +08:00
j2rong4cn
103abc942e refactor: pass api_url through context (#457) 
* refactor: pass `api_url` through context

* 移除 LinkArgs.HttpReq

* pref(alias): 减少不必要下载代理

* 修复bug

* net: 支持1并发 分片下载
 2025-06-30 15:48:05 +08:00
Kuingsmile
fdcc2f136e chore: change module name to OpenListTeam/OpenList (#2) 
* Enable blank issue

* chore(README.md): update docs (temporally)

* Update FUNDING.yml

* chore: purge README.md

* chore: change module name to OpenListTeam/OpenList

* fix: fix link errors

* chore: remove v3 in module name

* fix: resolve some conficts

* fix: resolve conficts

* docs: update with latest file

---------

Co-authored-by: ShenLin <773933146@qq.com>
Co-authored-by: Hantong Chen <cxwdyx620@gmail.com>
Co-authored-by: joshua <i@joshua.su>
Co-authored-by: Hantong Chen <70561268+cxw620@users.noreply.github.com>
 2025-06-12 22:02:46 +08:00
j2rong4cn
ddffacf07b perf: optimize IO read/write usage (#8243) 
* perf: optimize IO read/write usage

* .

* Update drivers/139/driver.go

Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
 2025-04-12 16:55:31 +08:00
j2rong4cn
a6304285b6 fix: revert "refactor(net): pass request header" (#8269) 
5be50e77d9
 2025-04-03 20:35:52 +08:00
KirCute
1335f80362 feat(archive): support multipart archives (#8184 close #8015) 
* feat(archive): multipart support & sevenzip tool

* feat(archive): rardecode tool

* feat(archive): support decompress multi-selected

* fix(archive): decompress response filter internal

* feat(archive): support multipart zip

* fix: more applicable AcceptedMultipartExtensions interface
 2025-03-27 23:20:44 +08:00
j2rong4cn
4145734c18 refactor(net): pass request header (#8031 close #8008) 
* refactor(net): pass request header

* feat(proxy): add `Etag` to response header

* refactor
 2025-03-01 18:35:34 +08:00
KirCute_ECT
3b71500f23 feat(traffic): support limit task worker count & file stream rate (#7948) 
* feat: set task workers num & client stream rate limit

* feat: server stream rate limit

* upgrade xhofe/tache

* .
 2025-02-16 12:22:11 +08:00
KirCute_ECT
779c293f04 fix(driver): implement canceling and updating progress for putting for some drivers (#7847) 
* fix(driver): additionally implement canceling and updating progress for putting for some drivers

* refactor: add driver archive api into template

* fix(123): use built-in MD5 to avoid caching full

* .

* fix build failed
 2025-02-01 17:29:55 +08:00
j2rong4cn
5c5d8378e5 fix(archive): unable to preview (#7843) 
* fix(archive): unrecognition zip

* feat(archive): add tree for zip meta

* fix bug

* refactor(archive):  meta cache time use Link Expiration first

* feat(archive): return sort policy in meta (#2)

* refactor

* perf(archive): reduce new network requests

---------

Co-authored-by: KirCute_ECT <951206789@qq.com>
 2025-01-27 20:08:56 +08:00
j2rong4cn
2be0c3d1a0 feat(alias): add DownloadConcurrency and DownloadPartSize option (#7829) 
* fix(net): goroutine logic bug (AlistGo/alist#7215)

* Fix goroutine logic bug

* Fix bug

---------

Co-authored-by: hpy hs <hshpy.pengyu@gmail.com>

* perf(net): sequential and dynamic concurrency

* fix(net): incorrect error return

* feat(alias):  add `DownloadConcurrency` and `DownloadPartSize` option

* feat(net): add `ConcurrencyLimit`

* pref(net): create `chunk` on demand

* refactor

* refactor

* fix(net): `r.Closers.Add` has no effect

* refactor

---------

Co-authored-by: hpy hs <hshpy.pengyu@gmail.com>
 2025-01-27 20:08:39 +08:00
KirCute_ECT
bb40e2e2cd feat(archive): archive manage (#7817) 
* feat(archive): archive management

* fix(ftp-server): remove duplicate ReadAtSeeker realization

* fix(archive): bad seeking of SeekableStream

* fix(archive): split internal and driver extraction api

* feat(archive): patch

* fix(shutdown): clear decompress upload tasks

* chore

* feat(archive): support .iso format

* chore
 2025-01-18 23:28:12 +08:00
seiuneko
8278d3875b fix: ignore os.ErrClosed error on repeated FileStream close operations (#6762) 
Also resolves the issue where S3 PutObject returns a 500 status code.
 2024-07-14 20:59:24 +08:00
Mmx
b95df1d745 perf: use io copy with buffer pool (#6389) 
* feat: add io methods with buffer

* chore: move io.Copy calls to utils.CopyWithBuffer
 2024-04-25 20:11:15 +08:00
NewbieOrange
e37465e67e feat(crypt): force stream upload for supported drivers (#6270) 2024-03-29 14:42:01 +08:00
foxxorcat
8ef89ad0a4 fix(baidu_netdisk): hash and error 2 (#5356) 
* fix(baidu):hash and error:2

* fix:invalid memory address
 2023-10-10 18:08:27 +08:00
Sean
b313ac4daa fix(crypt): fix 139cloud hack (#5178) 
(cherry picked from commit 18bf64af47e58cc69cdd2e598de9c19538a7bf78)
 2023-09-06 14:12:01 +08:00
fregie
b604e21c69 feat(webdav): support http chunked request (close #5161 in #5162) 
But we do not recommend not adding the content-length header when putting files
 2023-09-05 13:03:29 +08:00
Sean
e7c0d94b44 fix: form upload when ticked As A Task (#5145) 2023-09-03 15:40:40 +08:00
Andy Hsu
ce10c9f120 fix: temp file not close and incorrect WebPutAsTask 2023-08-28 18:18:02 +08:00
Sean
a3748af772 feat: misc improvements about upload/copy/hash (#5045) 
general: add createTime/updateTime support in webdav and some drivers
general: add hash support in some drivers
general: cross-storage rapid-upload support
general: enhance upload to avoid local temp file if possible
general: replace readseekcloser with File interface to speed upstream operations
feat(aliyun_open): same as above
feat(crypt): add hack for 139cloud

Close #4934 
Close #4819 

baidu_netdisk needs to improve the upload code to support rapid-upload
 2023-08-27 21:14:23 +08:00