> For the complete documentation index, see [llms.txt](https://sfdigitalservices.gitbook.io/data-publishing-process/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sfdigitalservices.gitbook.io/data-publishing-process/data-pipeline/pipeline-basics.md).

# What is a data pipeline

## What's a data pipeline?

To share your data with the public, we will need to move it from wherever it lives (your database, system, app, spreadsheet, etc.) onto our platform. We may also have to change columns or values before publishing it. The process of moving and cleaning data is a 'data pipeline'.

{% hint style="info" %}
The process of moving data is often called **ETL** because the steps are:

* **E**xtract data from it's source
* Perform **T**ransformations on the data
* **L**oad the data into it's target destination
  {% endhint %}

## Which pipeline is best for me?

The first question to ask yourself is, how often will this dataset update?

* If the answer is Yearly or Never, you could consider [manual publishing](/data-publishing-process/data-pipeline/pipeline-basics/manual-publishing.md)
* If the answer is more frequent than Quarterly, the process should be [automated](/data-publishing-process/data-pipeline/pipeline-basics/data-pipeline.md)

The next two pages cover manual and automated data pipelines.