Table of Contents
- The Challenge
- Architecture & Solution
- Tech Stack
- Key Engineering Decisions
- Core Features
- Security Implementation
- API Endpoints
- Frontend Components
- Deployment
- Roadmap
The Challenge
Rclone is a powerful command-line tool for managing cloud storage across 50+ providers, but its CLI-only interface presents significant barriers for teams and enterprises. Non-technical users struggle with complex syntax, multiple servers require separate SSH connections, and monitoring ongoing transfers requires constant terminal attention.
The fundamental challenge was creating a user-friendly web interface that could wrap Rclone's functionality while adding enterprise-grade features like multi-host management, scheduled jobs, real-time monitoring, and proper access controls — all without compromising the security of the underlying Rclone RC API.
The requirement: Build a production-grade web interface that provides secure, authenticated access to Rclone across multiple hosts, enables visual file management with dry-run safety, supports cron-based job scheduling, offers real-time transfer monitoring via WebSockets, and implements proper audit logging for compliance.
Architecture & Solution
The system follows a proxy architecture where a secure Node.js layer sits between users and the Rclone RC API:
Project Structure
rclone-manager/
├── backend/
│ ├── src/
│ │ ├── services/ # Core logic (Rclone, Auth, Audit, Scheduler)
│ │ ├── routes/ # API endpoints
│ │ ├── middleware/ # Security & validation
│ │ └── types/ # TypeScript definitions
│ └── tests/ # Jest unit tests
├── frontend-next/ # Next.js 14 web client
│ ├── app/ # Pages
│ ├── components/ # UI components
│ └── hooks/ # Custom hooks
├── docker-compose.yml
└── Dockerfile.prodTech Stack
| Layer | Technology | Version |
|---|---|---|
| Frontend | Next.js | 14.x |
| UI Library | shadcn/ui | latest |
| Styling | Tailwind CSS | 4.x |
| Backend | Express | 4.x |
| Runtime | Node.js | 20.x |
| Language | TypeScript | 5.x |
| Database | SQLite | 3.x |
| ORM | better-sqlite3 | 12.x |
| Core Engine | Rclone | 1.65+ |
| WebSocket | ws | 8.x |
| Security | helmet, bcrypt | 7.x/6.x |
| Scheduling | cron-parser | 5.x |
| Container | Docker | latest |
Key Engineering Decisions
1. Secure Rclone RC Proxy Layer
The backend acts as a secure proxy, ensuring no direct exposure of the Rclone RC port:
// backend/src/services/rclone.service.ts
export class RcloneService {
private async forwardRequest(hostId: string, endpoint: string, body?: any) {
const host = await this.getHostConfig(hostId);
// Validate host is accessible
const health = await this.checkHostHealth(host);
if (!health.online) {
throw new Error(`Host ${host.name} is offline`);
}
// Forward request to specific host's Rclone RC
const response = await axios.post(
`${host.rcUrl}/ ${endpoint}`,
body,
{
auth: {
username: host.rcUser,
password: host.rcPass,
},
timeout: 30000,
}
);
return response.data;
}
async listRemotes(hostId: string) {
return this.forwardRequest(hostId, '/config/listremotes');
}
async copyFile(hostId: string, srcRemote: string, dstRemote: string, srcPath: string, dstPath: string) {
return this.forwardRequest(hostId, '/operations/copyfile', {
srcFs: `${srcRemote}:${srcPath}`,
dstFs: `${dstRemote}:${dstPath}`,
});
}
}2. Session-Based Authentication with Role-Based Access
// backend/src/middleware/auth.middleware.ts
export function authenticate(req: Request, res: Response, next: NextFunction) {
if (!req.session.userId) {
return res.status(401).json({ error: 'Unauthorized' });
}
const user = userService.getById(req.session.userId);
if (!user || !user.isActive) {
return res.status(401).json({ error: 'User not found or inactive' });
}
req.user = user;
next();
}
export function authorize(...roles: string[]) {
return (req: Request, res: Response, next: NextFunction) => {
if (!roles.includes(req.user.role)) {
return res.status(403).json({ error: 'Forbidden' });
}
next();
};
}
// Usage
router.get('/hosts', authenticate, authorize('ADMIN'), listHosts);
router.get('/files', authenticate, listFiles); // Any authenticated user3. Real-Time Transfer Monitoring with WebSockets
// backend/src/services/stats.service.ts
export class StatsService {
private wsClients: Set<WebSocket> = new Set();
constructor() {
// Poll Rclone stats every second
setInterval(async () => {
const stats = await this.gatherStats();
this.broadcast(stats);
}, 1000);
}
private async gatherStats() {
const activeJobs = await rcloneService.getJobs();
const transfers = await rcloneService.getTransferStats();
const system = await this.getSystemResources();
return {
jobs: activeJobs,
transfers: transfers,
cpu: system.cpu,
memory: system.memory,
bandwidth: system.bandwidth,
};
}
private broadcast(data: any) {
this.wsClients.forEach(client => {
if (client.readyState === WebSocket.OPEN) {
client.send(JSON.stringify(data));
}
});
}
}4. Cron-Based Job Scheduling
// backend/src/services/scheduler.service.ts
export class SchedulerService {
async createSchedule(scheduleData: CreateScheduleDto) {
// Validate cron expression
const cron = cronParser.parseExpression(scheduleData.cronExpression);
const nextRun = cron.next().toDate();
const schedule = await db.schedules.create({
data: {
name: scheduleData.name,
hostId: scheduleData.hostId,
command: scheduleData.command,
sourceRemote: scheduleData.sourceRemote,
destinationRemote: scheduleData.destinationRemote,
cronExpression: scheduleData.cronExpression,
nextRunAt: nextRun,
isActive: true,
},
});
// Schedule the job
this.scheduleJob(schedule);
return schedule;
}
private scheduleJob(schedule: Schedule) {
const job = cron.schedule(schedule.cronExpression, async () => {
await this.executeJob(schedule);
// Update next run time
const next = cronParser.parseExpression(schedule.cronExpression).next();
await db.schedules.update({
where: { id: schedule.id },
data: { lastRunAt: new Date(), nextRunAt: next.toDate() },
});
});
this.jobs.set(schedule.id, job);
}
}5. Configuration Versioning and Backup
// backend/src/services/config.service.ts
export class ConfigService {
async backupConfig(hostId: string) {
const host = await db.hosts.findUnique({ where: { id: hostId }});
const config = await rcloneService.getConfigDump(host);
// Store backup with timestamp
const backup = await db.configBackups.create({
data: {
hostId,
configData: config,
backupType: 'MANUAL',
},
});
// Keep only last 10 backups
await this.pruneOldBackups(hostId, 10);
return backup;
}
async restoreConfig(backupId: string) {
const backup = await db.configBackups.findUnique({
where: { id: backupId }}
});
// Verify backup integrity before restore
if (!this.verifyConfigIntegrity(backup.configData)) {
throw new Error('Backup integrity check failed');
}
// Restore
await rcloneService.setConfigDump(backup.hostId, backup.configData);
return { success: true };
}
}Core Features
Multi-Host Management
| Feature | Description |
|---|---|
| Host Registry | Add multiple Rclone VPS/instances |
| Remote Cloning | Copy remote configs between hosts |
| Health Monitoring | Real-time status checks |
| Connection Profiles | Persistent connection settings |
File Browser
| Feature | Description |
|---|---|
| Directory Navigation | Browse remote files like local |
| Dry Run Mode | Simulate copy/move before execution |
| File Operations | Copy, move, delete, create directory |
| File Preview | View file metadata |
Mount Management
| Feature | Description |
|---|---|
| Mount Creation | Create FUSE mounts |
| Auto-Mount | Mount on system startup |
| Health Monitoring | Real-time mount status |
| Process Management | Auto-spawn and restart |
Job Control
| Feature | Description |
|---|---|
| Job Types | Copy, move, sync, sync backup |
| Control | Pause, resume, stop |
| Progress Tracking | Real-time speed and percentage |
| History | Complete job execution history |
Scheduling
| Feature | Description |
|---|---|
| Cron Expression | Full cron syntax support |
| Visual Editor | GUI without cron knowledge |
| Recurrence | Daily, weekly, monthly patterns |
| Execution History | Track past scheduled runs |
Security Implementation
Authentication & Authorization
| Feature | Implementation |
|---|---|
| Password Hashing | bcrypt with salt |
| Session Management | express-session with secure cookies |
| Role-Based Access | ADMIN and VIEWER roles |
| Host-Level Permissions | Users see only authorized hosts |
Audit Logging
// All critical actions are logged
const actions = [
'LOGIN', 'LOGOUT', 'FAILED_LOGIN',
'FILE_DELETE', 'FILE_COPY', 'FILE_MOVE',
'JOB_START', 'JOB_STOP', 'JOB_COMPLETE',
'CONFIG_CREATE', 'CONFIG_UPDATE', 'CONFIG_DELETE',
'HOST_CREATE', 'HOST_DELETE',
'SCHEDULE_CREATE', 'SCHEDULE_UPDATE', 'SCHEDULE_DELETE',
'MOUNT_CREATE', 'MOUNT_DELETE',
];
// Audit entry structure
{
action: string,
userId: string,
hostId: string,
details: string,
ipAddress: string,
userAgent: string,
timestamp: Date,
}Configuration Protection
- Automatic config versioning before any modification
- Encrypted backup to remote cloud storage
- Config integrity verification before restore
API Endpoints
Authentication
| Method | Endpoint | Description |
|---|---|---|
| POST | /auth/login | User login |
| POST | /auth/logout | User logout |
| GET | /auth/me | Current user info |
| POST | /auth/password | Change password |
Hosts
| Method | Endpoint | Description |
|---|---|---|
| GET | /hosts | List all hosts |
| POST | /hosts | Add new host |
| GET | /hosts/:id | Get host details |
| PATCH | /hosts/:id | Update host |
| DELETE | /hosts/:id | Remove host |
| POST | /hosts/:id/test | Test connection |
Remotes
| Method | Endpoint | Description |
|---|---|---|
| GET | /remotes/:hostId | List remotes |
| POST | /remotes/:hostId | Create remote |
| PATCH | /remotes/:hostId/:name | Update remote |
| DELETE | /remotes/:hostId/:name | Delete remote |
Files
| Method | Endpoint | Description |
|---|---|---|
| GET | /files/:hostId/:remote | List files |
| POST | /files/:hostId/copy | Copy file |
| POST | /files/:hostId/move | Move file |
| DELETE | /files/:hostId | Delete file |
| POST | /files/:hostId/mkdir | Create directory |
Mounts
| Method | Endpoint | Description |
|---|---|---|
| GET | /mounts/:hostId | List mounts |
| POST | /mounts/:hostId | Create mount |
| DELETE | /mounts/:hostId/:mount | Unmount |
Jobs
| Method | Endpoint | Description |
|---|---|---|
| GET | /jobs | List all jobs |
| GET | /jobs/:id | Job details |
| POST | /jobs/:id/pause | Pause job |
| POST | /jobs/:id/resume | Resume job |
| POST | /jobs/:id/stop | Stop job |
Schedules
| Method | Endpoint | Description |
|---|---|---|
| GET | /schedules | List schedules |
| POST | /schedules | Create schedule |
| PATCH | /schedules/:id | Update schedule |
| DELETE | /schedules/:id | Delete schedule |
Stats
| Method | Endpoint | Description |
|---|---|---|
| GET | /stats/transfer | Transfer stats |
| GET | /stats/system | System resources |
| GET | /stats/jobs | Job statistics |
| WS | /ws/stats | Real-time stats stream |
Frontend Components
The Next.js frontend provides a Windows Explorer-like interface:
| Component | Description |
|---|---|
| Dashboard | Overview with host status, recent jobs, stats |
| File Browser | Navigate and manage cloud files |
| Mounts | Manage FUSE mounts |
| Jobs | Monitor and control transfer jobs |
| Schedules | Create and manage cron jobs |
| Settings | User and system configuration |
Real-Time Features
- WebSocket connection for live transfer updates
- Speed, bandwidth, and progress monitoring
- Job state changes (pause/resume/complete)
- System resource monitoring
Deployment
Docker Production
# Configure environment
cp .env.prod.example .env
# Build and start
docker-compose -f docker-compose.prod.yml up -d --buildServices
| Service | Port | Description |
|---|---|---|
| Frontend | 3000 | Next.js web UI |
| Backend | 3001 | Express API |
| Rclone | 5572 | Rclone RC |
Configuration Variables
| Variable | Default | Description |
|---|---|---|
| ADMIN_PASSWORD | admin | Web UI password |
| SESSION_SECRET | random | Session encryption |
| RCLONE_API_URL | http://rclone:5572 | Rclone RC URL |
| RCLONE_API_USER | admin | Rclone RC user |
| RCLONE_API_PASS | (empty) | Rclone RC password |
Roadmap
- Phase 1 — Core Platform (Complete) - Auth, hosts, remotes
- Phase 2 — File Browser (Complete) - Navigation, copy, move, delete
- Phase 3 — Mount Management (Complete) - FUSE mounting, health checks
- Phase 4 — Job Control (Complete) - Background transfers, WebSocket
- Phase 5 — Scheduling (Complete) - Cron-based automation
- Phase 6 — Security (Complete) - Audit logging, config versioning
- Phase 7 — Advanced Features (In Progress) - Bandwidth limits, sync verification
- Phase 8 — Clustering (Planned) - Distributed job processing
- Phase 9 — Mobile App (Planned) - iOS/Android monitoring
Conclusion
The Aatmanova Cloud Manager transforms the powerful but CLI-only Rclone into an accessible enterprise web application. The secure proxy architecture ensures the Rclone RC API is never directly exposed while providing comprehensive multi-host management from a centralized dashboard.
Real-time WebSocket monitoring enables teams to track transfers as they happen, while cron-based scheduling automates recurring backup tasks without manual intervention. The SQLite-backed audit logging provides compliance-ready tracking of all critical actions, and the configuration versioning system protects against accidental misconfiguration.
The production-ready Docker deployment with optimized non-root containers ensures security in production environments, while the modular architecture supports future enhancements like distributed job processing and mobile applications.
Architecture Feedback
Spotted a potential optimization or antipattern? Let me know.