In this article, the author shares his first-person experience building a file-sharing service in Rust. He recalls how he used to use IRC, where there was no easy way to send images or documents, and explains why he decided to write his own minimalist tool, rustypaste. It is a service that allows you to upload files, shorten links, create disposable URLs, and set expiration dates. The author shows step-by-step the history of the project, its features, and its deployment in the cloud via Shuttle.rs to make the service publicly available. The article is written in the form of a personal narrative, which makes it lively and interesting even for those who are just getting acquainted with the technology.
rustypaste » is a minimal native file upload/pastebin service written in Rust. In this post, I will cover its features and deployment history on shuttle.rs to make it publicly available for free use.
To properly convey the idea behind file sharing, I need to tell you a story.
In the early days of the internet (although I didn’t live to see it), communication was simpler and, naturally, less secure. One of the things people used was the almighty Internet Relay Chat (IRC), a famous chat system that is still used by some projects today. Back then, people would create channels there and enjoy unprotected, less censored, and seemingly anonymous communication. Today, some people, including me, consider those days to be the golden age of the internet. Everything was so peaceful, and you could smell the dusty air coming from the old PC fan barely running Windows XP with an Intel Pentium 4 processor. *sigh* Happy times.
You couldn’t share images on IRC. At least it wasn’t that easy, since IRC clients were mostly text-based and designed to run in a terminal (the author is talking about BitchX here). Of course it wasn’t as easy and convenient as modern chat programs, and we can’t expect it to be. But I guess it’s still Cheff’s KISS, what is file sharing support anyway?
It requires storing files somewhere and transferring them to the recipient. There are so many things to consider, such as upload limits, nudity detection (if that’s important to you), privacy, file storage, additional server costs, disk space, etc. In short, file sharing is THOUGHT! Describing how to make chain mail in Minecraft to your friends using only text messages instead of sending a screenshot is MUCH BETTER. Or wait, we can use ASCII images, oh yeah, that’s possible too. Right?
I guess I’m a bit off topic. But you see where I’m getting at? There was no easy support for file sharing on IRC, so people had to come up with their own solutions, in other words, their own upload servers. During my time on IRC, I’ve met many users who use links like the one below just to share files/images:
https://paste.xinu.at/jHKy
https://0x0.st/H8hSa.png
In a way, in today’s world, doing this seems redundant and actually more complicated than just pressing 3 buttons on an interface. While I agree with that, I approach this situation from a different perspective.
Technology is, above all, about control. I think this is especially true today, as every corner of the human brain is easily subjugated by technology. We are deeply integrated with this world, and when we don’t control it, it controls us. Cliché, but true.
In the case of file sharing, having your own (self-hosted) service to upload your files is simply about controlling what you put out into this virtual world. What you want to share is there if you want it, and it’s there under the circumstances you choose. Maybe the files are being uploaded from an external drive on your home server. Maybe they expire in 2 hours. Maybe you want to see who clicked on the link. It all comes down to owning your data.
If you consider these points, it becomes clear that the lack of conditions of the IRC era revealed the perfect solution for file sharing, which complements the idea that centralization is bad. Ironically, the limitations of IRC led to the emergence of self-hosted file sharing.
Although I joined IRC quite late (3 years ago), I got used to it quite quickly. At first I used a few public file sharing services, but then I really liked the idea of self-hosting and decided to host my own upload server. That’s why I wrote the post “Start your own no-nonsense file hosting service with 0x0” 2 years ago. I used 0x0 for a while, but then I had to move to another server and I was too lazy to set it up again. Plus, it was written in Python x_x.
That day I decided to write my own Pastebin service. And of course, I was going to write it in Rust.
GitHub : https://github.com/orhun/rustypaste
Public instance : https://rustypaste.shuttleapp.rs
rustypaste is a minimal and simple upload service written in Rust and implemented using the Actix web framework. I chose Actix at the time because it was the most suitable framework in terms of security, performance and simplicity. As a regular user of rustypaste for almost 2 years now, I can say that it is in a stable state and has a bunch of features that support advanced customization.
The easiest way to interact with the rustypaste server is to use curl, but you can also use a command line tool called rpaste, which is written in Rust.
Command line tool : https://github.com/orhun/rustypaste-cli
Usage:
rpaste [options] <file(s)>
Options:
-h, --help prints help information
-v, --version prints version information
-V, --server-version
retrieves the server version
-o, --oneshot generates one shot links
-p, --pretty prettifies the output
-c, --config CONFIG sets the configuration file
-s, --server SERVER sets the address of the rustypaste server
-a, --auth TOKEN sets the authentication token
-u, --url URL sets the URL to shorten
-r, --remote URL sets the remote URL for uploading
-e, --expire TIME sets the expiration time for the link
Download file
curl -F "[email protected]" https://rustypaste.shuttleapp.rs rpaste ferris.txt
URL shortening
curl -F "url=https://example.com/some/long/url" https://rustypaste.shuttleapp.rs rpaste -u https://example.com/some/long/url
Download file from URL
curl -F "remote=https://example.com/file.png" https://rustypaste.shuttleapp.rs rpaste -r https://example.com/file.png
File expires in 10 minutes (also works for URLs)
curl -F "[email protected]" -H "expire:10min" https://rustypaste.shuttleapp.rs rpaste -e 10min ferris.txt
Delete file after viewing once
curl -F "[email protected]" https://rustypaste.shuttleapp.rs rpaste -o ferris.txt
Authenticate on the server
curl -F "[email protected]" -H "Authorization: <auth_token>" https://rustypaste.shuttleapp.rs rpaste -a "<auth_token>" ferris.txt(ви також можете використовувати файл конфігурації)
To configure rustypaste, you only need one configuration file. The default file can be viewed at репозиторії GitHub ( config.toml) .
Here are some things you can configure and change:
Random Filenames: Pet Name (capital-mosquito.txt). Alphanumeric String (yB84D2Dv.txt)
MIME Types: Supports override and blacklisting. Supports forced download via?download=true
Supports disabling duplicate downloads
Auto-expiration + auto-delete files
If you want to host your own RustyPaste server, there are a few ways to do it:
Run the single binary (preferably installed from your distribution’s package manager). rustypasteavailable у репозиторіях Arch Linux .
Use a lightweight Docker image
docker-compose.yml (example)
Конфігурація Nginx (example)
Shuttle is a cloud development platform built on Rust that lets you deploy your application while taking care of all your infrastructure.
Shuttle derives the infrastructure definition from the Rust code itself using function signatures and annotations. It’s a great service that makes cloud deployment incredibly easy and supports multiple Rust frameworks, including Actix. Plus, you can manage everything (deployment, monitoring, resource management, etc.) with a single cargo subcommand, run from the command line, cargo shuttle . Considering all this, it’s no surprise that they’re supported by YC:o
They currently support a hobby plan which is practically free, and you can contact them if the following features don’t suit/are not enough for you:
Unlimited Deployment
150K Requests Per Month
500MB Database Storage
That’s quite enough, rustypaste, so I decided to use Shuttle.
However, I had one question about storage space. So I put it to them. Discord:
orhun: How does Shuttle deployment store files? Are they loaded inside an isolated container? If so, are there any disk space/memory/CPU limitations?
jonaro00: Your project runs in a Docker container, but writing to the file system is not recommended because the container is cleaned up during this cargo shuttle project restart (which you do when upgrading a Shuttle version or when something breaks). For persistent storage, you need to use some form of database. You can check the supported databases in the documentation, but there are also others in development.
orhun: I plan to automatically delete files after about an hour. Do you recommend using a file system in this case?
jonaro00: Yes, for this use case, everything seems to work fine.
However, I’m not sure if there are any disk space limitations for deploying Shuttle, and we couldn’t find out on Discord, so let me know if you know anything about it. Now that we have enough information about Shuttle, let’s deploy the Rust project!
Start by installing cargo-shuttle and logging in:
$ cargo install cargo-shuttle $ cargo shuttle login
After this step, you can simply create a Rust project ready for deployment. To create an Actix template, we can do:
$ cargo shuttle init -t actix-web
Currently, cargo-shuttle supports the following templates: actix-web, axum, poem, poise, rocket, salvo, serenity, tide, thruster, tower, warpwarp
After creation, the project looks something like this:
#!/usr/bin/env rust-script
//! [dependencies]
//! shuttle-runtime = "0.16.0"
//! actix-web = "4.3.1"
//! shuttle-actix-web = "0.16.0"
//! tokio = "1.28.1"
use actix_web::{get, web::ServiceConfig};
use shuttle_actix_web::ShuttleActixWeb;
// Define a route for the root path ("/") that returns a string "Hello World!"
#[get("/")]
async fn hello_world() -> &'static str {
"Hello World!"
}
// The main entry point for the Shuttle runtime.
#[shuttle_runtime::main]
async fn actix_web(
) -> ShuttleActixWeb<impl FnOnce(&mut ServiceConfig) + Send + Clone + 'static> {
// Define a closure to configure the ServiceConfig of the HttpServer.
let config = move |cfg: &mut ServiceConfig| {
// Register the hello_world function as a service at the root path.
cfg.service(hello_world);
};
// Wrap the closure in a ShuttleActixWeb object and return it.
Ok(config.into())
}
To test this locally:
$ cargo shuttle run
To deploy, it’s as simple as running the following command:
$ cargo shuttle deploy
You can then view the deployment at <app>.shuttleapp.rs and view the logs via cargo shuttle logs.
In the case of rustypaste, I already have an Actix application, so I need to tweak it a bit for deployment. Mainly, I needed to change the main entry point of the application. I decided to add a function flag called “shuttle” to wrap the additional Shuttle dependencies and activate another entry point. I didn’t touch the rest of the code, which is great when there’s less effort.
[features]
default = []
shuttle = [
"dep:shuttle-actix-web",
"dep:shuttle-runtime",
"dep:shuttle-static-folder",
"dep:tokio",
]
[dependencies]
# other dependencies
# ...
shuttle-actix-web = { version = "0.16.0", optional = true }
shuttle-runtime = { version = "0.16.0", optional = true }
shuttle-static-folder = { version = "0.16.0", optional = true }
tokio = { version = "1.28.1", optional = true }
Shuttle can be activated right now with –features shuttle. Then I hit another hurdle.
You see, rustypaste needs a config.toml file to work. And when we deploy it as a single binary, there will be no configuration file next to it. To solve this problem, I used a static folder to deploy Shuttle and edited the application entry point as follows:
#[cfg(feature = "shuttle")]
#[shuttle_runtime::main]
async fn actix_web(
#[shuttle_static_folder::StaticFolder(folder = "shuttle")] static_folder: PathBuf,
) -> ShuttleActixWeb<impl FnOnce(&mut ServiceConfig) + Send + Clone + 'static>
So if I put the configuration file in “shuttle/config.toml”, it will be part of the deployment. Great.
Now everything seems good. Let’s deploy!
З :Wait, how do you enable the shuttle feature via cargo shuttle? In other words, can you pass –features shuttle to it?
A :At the time of writing this blog post, this is not possible. So I created this task: https://github.com/shuttle-hq/shuttle/issues/913
З : Oh great, is there any way to solve it?
A : Yes, I decided to temporarily enable the default “shuttle” feature in Cargo.toml when I want to deploy.
З : Great. Is that how it works?
В : Of course!
# make "shuttle" feature default
$ sed -i 's|default = \[\]|default = \["shuttle"\]|g' Cargo.toml
# deploy without running tests
$ cargo shuttle deploy --no-test
2023-05-16T13:56:37.027724537Z INFO Entering queued state
2023-05-16T13:56:37.036643575Z DEBUG hyper::client::connect::http: connecting to 10.99.82.119:8001
2023-05-16T13:56:37.039322882Z DEBUG hyper::client::connect::http: connected to 10.99.82.119:8001
2023-05-16T13:56:37.041805834Z DEBUG hyper::client::pool: pooling idle connection for ("http", gateway:8001)
2023-05-16T13:56:37.046562562Z INFO Entering building state
2023-05-16T13:57:08.734972530Z INFO Finished release [optimized] target(s) in 31.63s
2023-05-16T13:57:08.924988890Z INFO Entering built state
2023-05-16T13:57:08.925183822Z INFO Entering loading state
2023-05-16T13:57:08.933992618Z DEBUG shuttle_deployer::runtime_manager: Starting alpha runtime at: /opt/shuttle/shuttle-executables/abe6d63d-8a0f-4d59-9545-8979261889e4
2023-05-16T13:57:10.937504021Z INFO shuttle_proto::runtime: connecting runtime client
2023-05-16T13:57:10.937602528Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:18369
2023-05-16T13:57:10.940519024Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:18369
2023-05-16T13:57:10.942760924Z DEBUG {service.ready=true} tower::buffer::worker: processing request
2023-05-16T13:57:10.949727188Z DEBUG {service.ready=true} tower::buffer::worker: processing request
2023-05-16T13:57:10.954448150Z INFO shuttle_deployer::deployment::run: loading project from: /opt/shuttle/shuttle-executables/abe6d63d-8a0f-4d59-9545-8979261889e4
2023-05-16T13:57:10.956904512Z DEBUG shuttle_deployer::deployment::run: loading service
2023-05-16T13:57:10.961554512Z DEBUG {service.ready=true} tower::buffer::worker: processing request
2023-05-16T13:57:10.978349558Z INFO {success="true"} shuttle_deployer::deployment::run: loading response
These static folders can be accessed by rustypaste
╭─────────╮
│ Folders │
╞═════════╡
│ shuttle │
╰─────────╯
Service Name: rustypaste
Deployment ID: abe6d63d-8a0f-4d59-9545-8979261889e4
Status: running
Last Updated: 2023-05-16T10:57:10Z
Hooray, the service is now available at the link https://rustypaste.shuttleapp.rs !
As a bonus, I wanted to deploy this service when I push a new tag to the repository, so I created the following GitHub Actions workflow:
name: Deploy on Shuttle
on:
push:
branches:
- master
tags:
- "v*.*.*"
pull_request:
branches:
- master
# allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
build:
name: Build / Deploy
runs-on: ubuntu-22.04
steps:
- name: Checkout the repository
uses: actions/checkout@v3
- name: Install Rust
uses: actions-rs/toolchain@v1
with:
toolchain: stable
profile: minimal
override: true
- name: Install cargo-binstall
uses: taiki-e/install-action@cargo-binstall
- name: Install cargo-shuttle
run: cargo binstall -y cargo-shuttle
- name: Prepare for deployment
shell: bash
run: sed -i 's|default = \[\]|default = \["shuttle"\]|g' Cargo.toml
- name: Build
run: cargo build --locked --verbose
- name: Deploy
if: ${{ startsWith(github.event.ref, 'refs/tags/v') || github.event_name == 'workflow_dispatch' }}
run: |
cargo shuttle login --api-key ${{ secrets.SHUTTLE_TOKEN }}
cargo shuttle project restart
cargo shuttle deploy --allow-dirty --no-test
One thing to note: I thought it would be more convenient to install cargo-shuttle over cargo-binstall as it loads the pre-compiled binaries faster. And finally, here’s how I put it all together:https://github.com/orhun/rustypaste/commit/29ddef8
There are many ways to share files online, and I’m on the side of “you should host your own files.” It may seem like a hassle to host your own server and set everything up, but I think it’s worth it in the end. I mean, you can even share a very secret document as a one-time URL so that it disappears from the face of the internet after the first view.
And who can deny that using your own domain to share an image/file is super cool? In short, I think there are many advantages to self-hosting your upload server and owning the stuff you share with the rest of the world.
By the way, feel free to use a public instance of rustypaste and let me know if you like it. There are no guarantees of stability for Shuttle deployments, but it’s worth a try!
Happy uploading!