> For the complete documentation index, see [llms.txt](https://sfdigitalservices.gitbook.io/data-publishing-process/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sfdigitalservices.gitbook.io/data-publishing-process/metadata/intro-to-metadata.md).

# Dataset Metadata

Resources to help people use your data effectively and instructions for how to submit metadata for DataSF-published Production Data

## What is Metadata?

Metadata is "data that provides information about other data". Simply, metadata is the information you will include with your dataset when you publish it. It should help people understand how to use the dataset and how the dataset was made. It includes a "Dataset Description" (dataset metadata) and "Column Descriptions" (column metadata). 

This pages provides information on the [City's Metadata Standards](/data-publishing-process/metadata/intro-to-metadata/metadata-standards.md) and how to submit metadata for data published by DataSF on the Unified Data Platform or the Open Data Portal. 

{% hint style="info" %}
**Metadata can be added or updated via DataSF's Production Data Metadata Submission App on the Unified Data Platform (UDP).** [See instructions below](#adding-or-updating-metadata-for-production-data-on-the-unified-data-platform).
{% endhint %}

See an example of a completed metadata page on the open data portal [here](https://data.sfgov.org/Housing-and-Buildings/Building-Permits/i98e-djp9/about_data). 

You can also review [DataSF's full metadata standards](/data-publishing-process/metadata/intro-to-metadata/metadata-standards.md). 

If you need to submit a request to have DataSF publish new or update existing Production Data on the UDP, please [submit a Production Data Request on the UDP Knowledge Hub](https://sfgov1.sharepoint.com/sites/CCSF-UnifiedDataPlatform?web=1).

## Dataset Metadata

Dataset metadata is information about the dataset overall: how was the dataset created, how should the data be used, what does each row represent? Essentially it is a primer for anyone who is interested in using the data.

Dataset metadata should use the following template for the "Description" Section:

```
<strong>A. SUMMARY</strong>
[What is this dataset?  What purpose does it serve? 
Are there relevant links to your department's website? 
Try to make this description understandable by the general public.]

<strong>B. HOW THE DATASET IS CREATED</strong>
[Describe the business process that generates this data.]

<strong>C. UPDATE PROCESS</strong>
[What is the process for updating this dataset and how often will it happen?]

<strong>D. HOW TO USE THIS DATASET</strong>
[Describe what an analyst should know about this dataset to use appropriately,
 e.g. what to watch out for, what to filter, etc.]

<strong>E. RELATED DATASETS</strong>
[Please list any closely related datasets here with hyperlinks to the datasets.
 See below for hyperlink formatting]
 
...if internal...

<strong>F. KNOWN USES </strong>
[Who is using this dataset? what is the business process it is being used for?] 
[Please list them off using the unstructured list <li> html function]
 
 
 Hyperlink formatting:
 <u><a href="https://tinyurl.com/7fruttpx">Whatever text you want</a></u>

```

## Column Metadata

Column Metadata is information on each individual column in the dataset. Each column should have a short description of what the column represents and how to use it.

Column Metadata is especially important if the columns have abbreviated names such as **"*****dt"*** instead of ***"date"*** as it will ensure there is a human readable version of the variable. If possible, also link to any additional documentation on how the variable was created.

{% hint style="warning" %}
**There are three required columns in every dataset:**

* data\_as\_of (please specify if this is the at the row or dataset level)
* data\_loaded\_at (DataSF will usually add this into the dataset for you)
* a primary key

See [Metadata Standards](/data-publishing-process/metadata/intro-to-metadata/metadata-standards.md) to learn more
{% endhint %}

## Adding or updating metadata for Production Data on the Unified Data Platform

Data published by DataSF on the Unified Data Platform is usually referred to as "Production Data" or "DataSF data." High-quality metadata is critical for ensuring Production Data is used accurately and effectively.&#x20;

{% hint style="info" %}
**Metadata can be added or updated via** [**DataSF's Production Data Metadata Submission App**](https://app.snowflake.com/streamlit/ccsf/udp/#/apps/DATASF_APPS.DSF.DATASF_METADATA_ENTRY_APP) **on the Unified Data Platform (UDP).** Make sure you are logged into MyApps already before clicking the link above.
{% endhint %}

**Follow the instructions below to access the Metadata App in the UDP:**

* Log in to the UDP from [MyApps](https://myapps.sfgov.org/ccsfportal/dashboard) (click "CCSF-Snowflake" on the MyApps landing page)
* [Click this link to access the Metadata App directly](https://app.snowflake.com/streamlit/ccsf/udp/#/apps/DATASF_APPS.DSF.DATASF_METADATA_ENTRY_APP) or go to Apps (left-hand navigation bar) and then search for "Metadata"&#x20;
* Follow the instructions and prompts in the Metadata App to submit or update metadata for a Production Dataset published by DataSF on your department's behalf. &#x20;

If you have any issues, please open a Production Data Request from [DataSF's Unified Data Platform Knowledge Hub](https://sfgov1.sharepoint.com/sites/CCSF-UnifiedDataPlatform?web=1).&#x20;

If you have been asked to add or update metadata on the Internal Data Portal (data.staff.sf.gov), [please follow these instructions instead](#adding-or-updating-metadata-for-production-data-on-the-unified-data-platform).&#x20;

You can also use our [Metadata Template](https://docs.google.com/document/d/1ERawbA4k70RW3TsJ9UeVjS3IJRpyY3Ez11ot0CHn6sE/edit?usp=sharing) to put together draft metadata for the dataset you're hoping to publish via DataSF.