TaskFlow
Production-Grade Async Task Processing — Kubernetes, GitOps, and Real Observability.
Project
AI Task Processing Platform
Role
DevOps & Full-Stack Engineer
Company
Personal Project / Engineering Assignment
Frontend & Backend
Queue & Worker
Kubernetes & GitOps
CI/CD & Containers
Observability
There's a big difference between an app that "works" and an app that's ready for production. TaskFlow was built to close that gap — not just to show that I can wire up a MERN stack, but to demonstrate what a real production deployment looks like.
THE IDEA 💡
"It runs locally" is not the same as "it's production-ready"
The motivation was straightforward: most developer portfolios show apps deployed to Heroku or Vercel with a single config file and call it done. That's fine for demos, but it doesn't show how you'd actually handle real infrastructure — multiple environments, audit-able deployments, self-healing clusters, and scaling policies that respond to actual load rather than guesswork.
TaskFlow is the answer to "what does production-grade actually look like?" — with every architecture decision documented and justified, not just assembled and shipped.
THE PROBLEM 🎯
Synchronous APIs collapse under burst traffic
The most common mistake in task processing is building it synchronously — the API receives a request, does the work, and returns a response. That design collapses quickly. When traffic spikes from 1 task per second to 50 tasks per second, your API becomes the bottleneck, response times climb, and eventually jobs start getting dropped.
The real engineering challenge is designing a system where the API is never the bottleneck, workers can scale completely independently, and the queue absorbs spikes without losing a single job. Then layered on top of that, you need infrastructure that delivers itself — changes to Kubernetes manifests applied automatically, manual cluster edits reverted, environment configs managed declaratively, and images built and published without anyone touching a command line.
THE SOLUTION 🚀
The API enqueues. The workers execute. The cluster delivers itself.
The key design decision is that the backend never actually processes a task. When a user submits a job, the backend validates the JWT, writes a pending document to MongoDB, pushes the task ID to a Redis list with a single RPUSH call — which takes under a millisecond — and immediately returns 202 Accepted. The actual work happens entirely in the Python worker, which runs a blocking BLPOP loop, picks up one job at a time, processes it in-process, and writes the result and logs back to MongoDB. The frontend polls for status every three seconds until completion. This means the API's response time is bounded only by a MongoDB write and a Redis push — both sub-10ms — regardless of how compute-heavy the actual task is.
Async Queue via Redis BLPOP
The backend enqueues and returns immediately. Workers drain the queue at their own pace. Redis absorbs any burst — during a 1,000-task spike, every job lands in MongoDB and the queue within 5 seconds, and workers drain it in roughly 10 more.
Kubernetes with Kustomize Environment Overlays
Three fully distinct environment configs — local, staging, production — each with their own replica counts, resource limits, and settings. No environment-specific manual steps, ever.
GitOps Delivery via Argo CD
Infrastructure changes are code changes. Argo CD auto-syncs the cluster to the infra repo, prunes deleted resources, and reverts manual changes. The cluster always matches what's in Git.
Horizontal Pod Autoscaler on Workers
In production, the HPA scales workers from 2 to 50 replicas based on CPU utilisation, with a 5-minute cooldown to prevent flapping. Workers are stateless by design, so scaling is always safe.
Redis-Backed Rate Limiting Across Replicas
Global limits (2,000 req/60s per IP) and auth-specific limits (10 req/60s per IP) are stored in Redis, so they work correctly across multiple backend pods — not just a single instance.
Live Log Streaming & Real-Time Status Tracking
The frontend streams task logs and polls for status updates as the worker runs, giving users live visibility into what's happening at each step — not just a spinner and a result.
GitHub Actions CI/CD Pipeline
Parallel build jobs per service. Multi-stage Dockerfiles produce slim, non-root images. Every push to main builds, tags, and pushes to Docker Hub automatically.
ARCHITECTURE 🏗️
Two flows that work independently: code delivery and task processing
It's worth separating the two distinct flows in TaskFlow because they solve different problems.
The delivery flow is how code gets from your laptop to the running cluster. A developer pushes to the application GitHub repository. GitHub Actions kicks off parallel build jobs — one per service (frontend, backend, worker) — producing slim multi-stage Docker images that get pushed to Docker Hub. Kubernetes manifests live in a completely separate infrastructure repository. Argo CD watches that repo and automatically applies any changes to the cluster. With prune: true, resources deleted from Git are removed from the cluster. With selfHeal: true, any manual change made directly to the cluster is reverted. The cluster always reflects exactly what's in Git — no drift, no surprises.
The task processing flow is how a user's job goes from form submission to completed result. The user submits a task through the Next.js frontend, which hits the backend via the gateway. The backend validates, writes a pending document to MongoDB, and pushes the task ID to a Redis list — done in under 10ms. The Python worker is running a blocking BLPOP loop; it picks up the task ID, marks the job as running in MongoDB with a timestamp and worker PID, executes the text operation in-process, and writes the result and final log entry back to MongoDB. The frontend polls every 3 seconds and updates the UI as soon as status reaches success or failed.
Project Gallery
A visual walkthrough of the interface and key screens.
