# What is a data pipeline

## What's a data pipeline?

To share your data with the public, we will need to move it from wherever it lives (your database, system, app, spreadsheet, etc.) onto our platform. We may also have to change columns or values before publishing it. The process of moving and cleaning data is a 'data pipeline'.

{% hint style="info" %}
The process of moving data is often called **ETL** because the steps are:

* **E**xtract data from it's source
* Perform **T**ransformations on the data
* **L**oad the data into it's target destination
  {% endhint %}

## Which pipeline is best for me?

The first question to ask yourself is, how often will this dataset update?

* If the answer is Yearly or Never, you could consider [manual publishing](/data-publishing-process/data-pipeline/pipeline-basics/manual-publishing.md)
* If the answer is more frequent than Quarterly, the process should be [automated](/data-publishing-process/data-pipeline/pipeline-basics/data-pipeline.md)

The next two pages cover manual and automated data pipelines.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://sfdigitalservices.gitbook.io/data-publishing-process/data-pipeline/pipeline-basics.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.