dailyburn ratchet

Ratchet

A library for performing data pipeline / ETL tasks in Go.

github

docs

Tutorial

Ratchet consists of a Pipeline with a series of PipelineStages, which each handle one or more DataProcessors.

Data Pipeline

A series of stages connected by channels, where each stage is a group of goroutines running the same function. In each stage, the goroutines receive values from upstream via inbound channels; perform some function on that data, usually producing new values; and send values downstream via outbound channels.

DataProcessors

Runs in their own goroutine so all the data is processed concurrently.

The DataProcessors send and receive JSON for convenience.

Ratchet gives you some useful data processors and an interface for you to implement custom ones.

Use tool to generate structs json2go

dailyburn/ratchet/data

Provides wrapper functions for json.Unmarshal() and json.Marshal() called data.ParseJSON() and data.NewJSON() that provide additional logging.