Jun 2025
Market Data Microservice
ShippedRole Solo
A containerized pipeline that fetches stock prices, streams them through Kafka, and stores a moving average for each symbol.
Stack: Python · FastAPI · Kafka · PostgreSQL · SQLAlchemy · Alembic · Docker
Four days, June 2025
The problem
A client wants the latest price for a symbol and a number derived from recent prices, without every request hitting an outside provider. The service answers GET /prices/latest from the database and calls the provider only on a cache miss. It stores the raw response, then publishes a price event to Kafka. A separate consumer reads those events and writes a 5-point moving average per symbol. POST /prices/poll schedules recurring fetches for a list of symbols.
The hardest stretch
Getting independent processes to start and agree. The API, the poller, the moving-average consumer, Kafka, and Postgres all start together under Docker Compose, and none of them is ready when the others come up. The database initializer retries with a delay until Postgres answers. The consumer waits for Kafka before it subscribes. Schema changes go through Alembic migrations instead of being created ad hoc.
Most of the bugs in this project were about ordering and readiness, not about the average itself.
Decisions
Chose Keep every raw provider response in its own table, linked from each price point
Rejected Storing only the normalized price
The original payload is the source of truth. If I change how a price is parsed, I can reprocess it instead of losing data.
Chose An abstract provider interface with Yahoo Finance as one implementation
Rejected Calling Yahoo directly from the route handler
The route does not know which provider it is talking to, so adding another one touches one new class.
Chose A separate Kafka consumer computes the average
Rejected Computing it inside the request
Ingest and analytics can fail and restart independently, and the request stays fast. At this size computing inline would have been simpler, so I count this as a trade I made on purpose.