A small data pipeline with serious habits
How incremental syncs, raw records, clean tables, and simple serverless tools can create reliable analytics.
A data pipeline does not need a large cluster to be production ready. For a daily Stripe sync, a scheduled function and a well-shaped warehouse can be cheaper, clearer, and easier to operate.
Small infrastructure can still use grown-up ideas: checkpoints, audit trails, least privilege, and repeatable setup.
Ahmad Ali, One Percent
flowchart LR
S[Cloud Scheduler] --> F[Sync function]
K[Secret Manager] --> F
F --> A[Stripe API]
A --> R[Raw tables]
R --> P[Processed tables]
P --> U[Unified customer view]
U --> B[BI snapshot]
F --> M[Sync history] Sync only what changed
A full copy is easy to understand, but it becomes wasteful as data grows. An incremental sync remembers the last successful checkpoint and asks for records created or updated after it.
The checkpoint should move only after a successful run. If a job fails halfway through, the next run can safely repeat work without creating missing gaps.
Keep raw data before making it tidy
Clean analytics tables are pleasant to query, but transformations can be wrong and source fields can change. Keeping the original JSON creates an audit trail and makes future reprocessing possible.
The Scrape-Stripe pipeline used separate raw, processed, metadata, unified, and BI layers. This sounds formal, but each layer answers a simple question.
- Raw: what did the source send?
- Processed: what fields are useful to query?
- Metadata: when did each sync run and did it succeed?
- Unified: how do records from different systems match?
- BI: what should a dashboard read quickly?
Serverless works when the job has edges
A daily billing sync has a clear start, a clear finish, and modest volume. That makes scheduled functions a good fit. Longer optional work can run as a separate job, then call the short function to finish shared steps.
The benefit is operational simplicity. There is no server waiting all day for a few minutes of work, and the typical small-business cost can stay below one dollar a month under the documented volume assumptions.
Make deployment boring
Secrets belong in a secret manager, not in source code. A dedicated service account should have only the access the job needs. Setup scripts should be safe to run again and should create the same result each time.
These habits are not extra ceremony. They make a small system recoverable by someone other than its original author. That is the difference between a script and a dependable product.
Ahmad Ali
Co-founder and CTO at One Percent. Writes about the practical choices behind products that need to work in the real world.