Loading...
Loading...
Ingestion pipeline architecture overview and convention reference. Use when you need a quick orientation to the pipeline framework or want to know which doctor agent to use for a specific concern.
npx skill4agent add posthog/posthog ingestion-pipeline-doctor-nodejsKafka message
→ messageAware()
→ parse headers/body
→ sequentially() for preprocessing
→ filterMap() to enrich context (e.g., team lookup)
→ teamAware()
→ groupBy(token:distinctId)
→ concurrently() for per-entity processing
→ gather()
→ pipeBatch() for batch operations
→ handleIngestionWarnings()
→ handleResults()
→ handleSideEffects()
→ build()nodejs/src/ingestion/analytics/joined-ingestion-pipeline.ts| What | Where |
|---|---|
| Step type | |
| Result types | |
| Doc-test chapters | |
| Joined pipeline | |
| Doctor agents | |
| Test helpers | |
| Concern | Agent | When to use |
|---|---|---|
| Step structure | | Factory pattern, type extension, config injection, naming |
| Result handling | | ok/dlq/drop/redirect, side effects, ingestion warnings |
| Composition | | Builder chain, concurrency, grouping, branching, retries |
| Testing | | Test helpers, assertions, fake timers, doc-test style |
<T extends Input>anyok()dlq()drop()redirect()ok(value, [effects])messageAwarehandleResultsmessageAwarehandleSideEffectsgroupByconcurrentlygatherconsumeAll()collectBatches()any