kernelci-storage update
Denys Fedoryshchenko <[email protected]> Wed, 19 Mar 2025 19:29:13 +0200
| Newsgroups | dev.linux.lists.kernelci |
|---|---|
| Message-ID | <[email protected]> |
(unnamed)
(text/markdown, 2.4 KB)
# KernelCI-Storage Project Update ## Deployment Status As mentioned in the last newsletter, we have migrated the production server to a new distribution (Bookworm) and deployed kernelci-storage. The system is also deployed on staging and is currently migrating to files-staging.kernelci.org. ## Implementation Details As planned in the first newsletter, we implemented a minimal API sufficient for the KernelCI project to work. Based on feedback, we updated endpoints to make them compatible with the kernelci-core storage backend implementation. Currently, kernelci-storage is written in Rust. The source code is available at the [kernelci-storage]([https://github.com/nuclearcat/kernelci-storage](https://github.com/nuclearcat/kernelci-storage)) repository but will be moved to the KernelCI organization namespace once pending issues are resolved. Available endpoints are documented in the [kernelci-storage README]([https://github.com/nuclearcat/kernelci-storage/blob/main/README.md](https://github.com/nuclearcat/kernelci-storage/blob/main/README.md)): - `GET|HEAD /path/to/file` - Download artifact (doesn't require authentication) or retrieve file info - `POST /upload` - Upload artifact (requires authentication), multipart/form-data - field: file0 - file to upload - field: path - path to save file to, can be absolute or relative to upload directory (default: upload directory root) ## Implemented Features - Heuristics to set correct content-type for uploaded files, while also supporting custom content-type settings - Support for uploading large files (resolving issues that existed in the Python backend) - Operation as a caching forward proxy to reduce charges from the storage provider per request - Support for downloading files with range requests ## Upcoming Features - Support for restricting upload directory (by user) - Lifecycle management for uploaded files (certain types of files might be deleted earlier) ## Challenges We has planned file browsing feature, but during implementation we faced challenge: Azure Blob Storage object list retrieval is very slow, 5000 objects (one request limit) takes 2-5 seconds to complete, and due dtb files we have tens of millions of them. We have solution for this, but it is not implemented yet. ## Future Plans We need your feedback! Please share your thoughts and suggestions on the mailing list or send message to our Discord server. Thanks for reading!