Skip to content

Limit the size of served files #834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jun 15, 2020
Merged

Limit the size of served files #834

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
e864615
sized_buffer: add buffer that limits the size of its content
pietroalbini Jun 12, 2020
2ef20c3
storage: require a maximum size when retrieving a file
pietroalbini Jun 12, 2020
c7d58a9
storage: add support for compressing with a custom algorithm
pietroalbini Jun 12, 2020
8a7fd3e
storage: ensure the decompressed content doesn't exceed the max size
pietroalbini Jun 12, 2020
d0f5708
storage: ensure downloads from s3 don't exceed the max size
pietroalbini Jun 12, 2020
4560081
storage: ensure downloads from the database don't exceed the max size
pietroalbini Jun 12, 2020
3d6a029
storage: cover more cases in ctest_decompression_too_big
pietroalbini Jun 15, 2020
6914673
storage: reduce number of database queries
pietroalbini Jun 15, 2020
95ed95d
sized_buffer: use reserve instead of reserve_exact when possible
pietroalbini Jun 15, 2020
8b07025
set max size to 5MB for html and 50MB for the rest
pietroalbini Jun 15, 2020
5f6b960
add global configuration for docs.rs
pietroalbini Jun 15, 2020
662930b
web: add test for getting files too big
pietroalbini Jun 15, 2020
7d56781
address review nits
pietroalbini Jun 15, 2020
7e37e40
test: update how the configuration is overridden
pietroalbini Jun 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/db/file.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ use std::path::{Path, PathBuf};

pub(crate) use crate::storage::Blob;

pub(crate) fn get_path(conn: &Connection, path: &str) -> Result<Blob> {
Storage::new(conn).get(path)
pub(crate) fn get_path(conn: &Connection, path: &str, max_size: usize) -> Result<Blob> {
Storage::new(conn).get(path, max_size)
}

/// Store all files in a directory and return [[mimetype, filename]] as Json
Expand Down
8 changes: 4 additions & 4 deletions src/storage/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ impl<'a> Storage<'a> {
DatabaseBackend::new(conn).into()
}
}
pub(crate) fn get(&self, path: &str) -> Result<Blob, Error> {
pub(crate) fn get(&self, path: &str, max_size: usize) -> Result<Blob, Error> {
let mut blob = match self {
Self::Database(db) => db.get(path),
Self::S3(s3) => s3.get(path),
Expand Down Expand Up @@ -282,7 +282,7 @@ mod test {
let name = Path::new(&blob.path);
assert!(stored_files.contains_key(name));

let actual = backend.get(&blob.path).unwrap();
let actual = backend.get(&blob.path, std::usize::MAX).unwrap();
assert_blob_eq(blob, &actual);
}

Expand Down Expand Up @@ -324,12 +324,12 @@ mod test {
"text/rust"
);

let file = backend.get("rustdoc/Cargo.toml").unwrap();
let file = backend.get("rustdoc/Cargo.toml", std::usize::MAX).unwrap();
assert_eq!(file.content, b"data");
assert_eq!(file.mime, "text/toml");
assert_eq!(file.path, "rustdoc/Cargo.toml");

let file = backend.get("rustdoc/src/main.rs").unwrap();
let file = backend.get("rustdoc/src/main.rs", std::usize::MAX).unwrap();
assert_eq!(file.content, b"data");
assert_eq!(file.mime, "text/rust");
assert_eq!(file.path, "rustdoc/src/main.rs");
Expand Down
4 changes: 3 additions & 1 deletion src/web/file.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,14 @@ use crate::{db, error::Result};
use iron::{status, Handler, IronError, IronResult, Request, Response};
use postgres::Connection;

const MAX_FILE_SIZE: usize = 5 * 1024 * 1024; // 5MB

pub(crate) struct File(pub(crate) db::file::Blob);

impl File {
/// Gets file from database
pub fn from_path(conn: &Connection, path: &str) -> Result<File> {
Ok(File(db::file::get_path(conn, path)?))
Ok(File(db::file::get_path(conn, path, MAX_FILE_SIZE)?))
}

/// Consumes File and creates a iron response
Expand Down