May 2026 - Jun 2026

TaskFlow

An infrastructure-focused job-processing system built around independently deployed API and worker services. Redis connects the services, while Docker, Kubernetes, Kustomize and Argo CD make the deployment inspectable. A small task interface helps trace jobs through the system and examine worker behavior.

IINNFFRRAA SSOOUURRCCEE

PROJECT: Cloud-native job infrastructure
FOCUS: Kubernetes, worker scaling and GitOps
APPLICATION: Next.js, Node.js, Python
INFRASTRUCTURE: Redis, MongoDB, Docker, Kubernetes

Overview

TaskFlow is an infrastructure project for deploying and operating a queued workload. The API, Python workers and data services run as separate components, making their deployment, replica counts and failure boundaries visible in Kubernetes.

I built the services and their Docker, Kubernetes and Kustomize configuration, with Argo CD used to inspect deployment state. The small text-processing jobs are a test workload: the main focus is how requests reach workers, how worker replicas share the queue, and how operators trace execution through logs and cluster resources.

TaskFlow overview — TaskFlow product overview.

Why I built it

I wanted to understand the operational path from a container image to a running workload: service boundaries, deployment configuration, worker replicas and the cluster resources that connect them. The task interface exists to exercise that infrastructure.

Using small text operations made it easier to inspect the infrastructure. The same job has an input, a place in the queue, a running worker and a final record, with failure states that can be examined separately.

The problem

A web request should not have to stay open while a job waits behind other work. The application needs to accept the task, give it an identity and report what happens later.

Separating the worker also creates failure boundaries. The API can be healthy while the queue is unavailable, or a worker can stop after taking a job. A useful task record needs to make those states visible.

The solution

The Node.js backend stores a task in MongoDB and places its payload on a Redis list. A Python worker uses BLPOP to wait for work without continuously polling an empty queue.

The worker moves the task through pending, running and success or failed states. It records timestamps, duration and log entries alongside the output. The interface can then show the job's progress independently of the submission request.

Independent worker replicas
Workers share the Redis queue and can be replicated separately from the API. More workers increase processing concurrency, but throughput still depends on job cost and shared Redis and MongoDB capacity.
Shared request limits
The API's rate limiter uses Redis-backed counters instead of keeping all request counts inside one application replica.
Inspectable failures
Failed tasks retain an error message and execution logs in their persistent record.

TaskFlow queue and service architecture — Service boundaries between the API, queue, workers and persistent task records.

The asynchronous job path

Node.js API
A submitted task gets a persistent MongoDB record.
Redis queue
The job payload waits on a list for a worker.
Python worker
BLPOP retrieves the task and the worker executes its operation.
Result and logs
MongoDB records success or failure for the interface to display.

Deployment

Docker Compose runs the services together locally. A separate infrastructure repository defines Kubernetes deployments for the application and workers, with Redis and MongoDB storage, services and a local ingress.

Kustomize separates the shared manifests from environment configuration, and the repository includes an Argo CD setup guide. The local deployment is the documented working path; staging and production overlays are described as placeholders.

TaskFlow deployment resource tree in Argo CD — Deployment resources and their relationships in Argo CD.

TaskFlow worker pod details in Argo CD — Inspecting a worker pod independently of the API deployment.

Outcome and queue design

BLPOP removes a job from the queue when a worker receives it. If that worker stops before recording completion, the queue does not automatically return the job to another worker. Database retries do not close that delivery gap.

The delivered workflow connects task submission, queued execution and stored results with timestamped logs. The interface exposes that lifecycle, while Docker Compose and the local Kubernetes configuration provide repeatable ways to run the services together.

Demo