31:T5eb,In previous guides, we've: 1. [_set up our Framework project_](https://observablehq.com/@rk2546/guide-local_observable_setup), 2. [_covered project structure_](https://observablehq.com/@rk2546/guide-project_structure), 3. [_imported functions and modules from NPM and Observable_](https://observablehq.com/@rk2546/guide-imports), and 4. [_styled our project_](https://observablehq.com/@rk2546/guide-styling_themes). In this guide, we'll cover how to use **data loaders** to import data into your _Framework_ project. _Framework_ projects give you a lot of flexibility with how data can be loaded - namely, in how it lets you use languages such as [_Python_](https://www.python.org/), [_R_](https://www.r-project.org/), [_Java_](https://www.java.com/en/), or [_PHP_](https://www.php.net/) instead of pure JavaScript. ${comment(htl.html`This guide will only cover JavaScript and Python-based data loaders. If you prefer other languages, you will have to look for tutorials online.`)} For this guide, I've created a new Markdown file called "data_loaders.md" inside of my `src/` directory. The page uses the "air" light theme. Here's how my `src/` file structure looks like now:

${render_filetree({
  "src/": {
      "index.md": null,
      "iris.md": null,
      "quotes.md": null,
      "notebook.md": null,
      "guide.md": null,
      "styling.md": null,
      "data_loaders.md": null,
      "components/": { "unique.js": null, "quotes.js": null },
      "data": { "iris.csv": null },
    },
})}

python --version

You should also double-check that you can safely call this shell command:

python3

#### If you do NOT have Python v3.+ installed: ##### Windows users: You can install Python using either [_Windows Subsystem for Linux (WSL)_](https://learn.microsoft.com/en-us/windows/python/web-frameworks) (the recommended way, as it's compatible with VS Code) or via an [_official installer_](https://www.python.org/downloads/). If using an official installer, make sure to add Python as an environment variable. You can also use [_pyenv_](https://github.com/pyenv/pyenv?tab=readme-ov-file#windows) if either do not work. ##### Mac OSX users: There are several ways to install (and manage) Python on OSX. Recommended ways include either [_uv_](https://docs.astral.sh/uv/getting-started/installation/) or [_pyenv_](https://github.com/pyenv/pyenv). Follow [_this guide_](https://mac.install.guide/python/install-pyenv) which covers both approaches. ##### Once installed: Double-check that you can safely call the following command via a shell command:

python3

${comment(html`If you realize that you have Python v3.+ installed but you cannot call python3, then you need to create a "symbolic link" from python3.exe to python.exe. To do this, follow these steps:

Run "cmd.exe"
cd to your Python's installation location; in most cases, it'll be located in:
\\Users\\[USERNAME]\\AppData\\Local\\Programs\\Python\\
Create the symbolic link by running the following command:
mklink python3.exe python.exe
If successful, you should see a new file called "python3.exe" created inside of the directory.

`, "NOTE (Windows Users Only)")}34:T419,### Step 2: Create and activate a Python virtual environment We want to create a "virtual environment" that lets us safely use Python while also running and/or building our _Framework_ project. This "virtual environment" is isolated from the rest of your system, meaning you can install packages and such without accidentally affecting your pre-existing setup (it's also generally just good practice anyways). To do this, run this command while you are CD'ed inside the root directory of your _Framework_ project:

python3 -m venv .venv

This creates an invisible folder called `.venv/`. Once done, then activate your new environment:

# Mac OSX:
source .venv/bin/activate
# Windows:
.venv\Scripts\activate

${comment(html`The new .venv/ directory is purely meant to be local and not shared on Github. To prevent it from being uploaded to Github when you're ready to push changes, make sure to modify your .gitignore:

.venv/*

`)}35:T433,### Step 3: Create your Python Scripts Now it's time to write your Python code and process our penguin data. When creating a [_JavaScript Loader_](#cell-70), we had to name our script file with a specific format. The same goes here for Python:

# Generic format
[DATASET_NAME].[EXPECTED FILE EXTENSION].py

# Example:
penguin_logistics.csv.py

Make sure to place this somewhere in the `src/` directly, preferably within a `data/` sub-directory. In our example, I created a Python script called `penguin_logistics.csv.py`. Note that I've also saved `penguins.csv` inside the same directory too. Our example file structure is now:

${render_filetree({
  "src/": {
      "index.md": null,
      "iris.md": null,
      "quotes.md": null,
      "notebook.md": null,
      "guide.md": null,
      "styling.md": null,
      "data_loaders.md": null,
      "components/": { "unique.js": null, "quotes.js": null },
      "data": { "iris.csv": null, "forecast.json.js": null, "penguin_logistics.csv.py": null, "penguins.csv": null },
    },
})}

36:T823,### Step 4: Write your Python script This one is up to you. Here's an example of our script for handling the penguin data, which uses [_Pandas_](https://pandas.pydata.org/) and [_scikit-learn_](https://scikit-learn.org/). Note the last line of the example script, which outputs the Pandas DataFrame to`sys.stdout`:

# Import libraries (must be installed)
import pandas as pd
from sklearn.linear_model import LogisticRegression
import sys

# Data access, wrangling and analysis
df = pd.read_csv("src/data/penguins.csv")
df_complete = df.dropna(
    subset=["culmen_length_mm", "culmen_depth_mm", "flipper_length_mm", "body_mass_g"]
)

X = df_complete.iloc[:, [2, 3, 4, 5]]
Y = df_complete.iloc[:, 0]

logreg = LogisticRegression()
logreg.fit(X, Y)

results = df_complete.copy()
results["predicted_species"] = logreg.predict(X)

df_out = df.merge(
    results[["predicted_species"]], how="left", left_index=True, right_index=True
)

# Write the data frame to CSV, and to standard output
df_out.to_csv(sys.stdout)

#### About Python Packages and _Framework_ _Framework_ inherently is a _JavaScript_ ecosystem, not a Python one. If you want to use Python packages, then you need to install them separately. This is typically done using [_pip_](https://pypi.org/project/pip/), which is auto-installed alongside Python after version 3.4 (similar to _npm_ with _Node.js_). Normally, you could install packages like Pandas with the following command:

pip install pandas

However, to make our lives easier, we can **create a `requirements.txt` file inside the project's root directory** that records all the Python packages that we need to install. Then, if we share this project with someone else or want to work on a different device, then we can simply call the following command:

pip install -r requirements.txt

For our penguin analysis, we need _Pandas_ and _scikit-learn_. So we created our own `requirements.txt` file that contains the following:

pandas
scikit-learn

37:T441,## What Next? The next guide we have is [_deploying your dashboards to Github Pages_](https://observablehq.com/@rk2546/guide-deployment), which allows you to share your projects as standalone websites. Alternatively, check out our other guides with _Framework_: ${framework_guides} ### Alternative Resources - [Documentation: Observable's guide on data loaders](https://observablehq.observablehq.cloud/framework/data-loaders) - [Video: "Write your first data loader in an Observable Framework project" by Observable](https://www.youtube.com/watch?v=HtlNctuB8ho) - [Examples of Data Loaders](https://github.com/observablehq/data-loader-examples) - [Symbolic Links with Python3 in Windows](https://stackoverflow.com/questions/39910730/python3-is-not-recognized-as-an-internal-or-external-command-operable-program) - [Using `mklink`](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/mklink) - [You can only use `cmd.exe` to use `mklink`](https://learn.microsoft.com/en-us/answers/questions/2821724/mklink-not-recognized?forum=windows-all&referrer=answers)38:[{"id":"0e2dadcebbba3a7ac21829a001d3d2b8e016af1dc3ce1dd497d096032bb7a14d06acf9523e1e6319195a37b29e6da88ac7f8c233825263b524f1dfa9d4bca5cd","href":"https://static.observableusercontent.com/files/0e2dadcebbba3a7ac21829a001d3d2b8e016af1dc3ce1dd497d096032bb7a14d06acf9523e1e6319195a37b29e6da88ac7f8c233825263b524f1dfa9d4bca5cd","name":"js_loader.png","type":"image/png","lastModified":1760912588266.746,"size":53234},{"id":"7f63959df02587fa2661dbeada66ee67b5218cc752777f62c1c4aa06c73c80ff45bc1f7e9409a693f7f8ccc56aa91adb5ce6615b7bfb1e4052380dd502163288","href":"https://static.observableusercontent.com/files/7f63959df02587fa2661dbeada66ee67b5218cc752777f62c1c4aa06c73c80ff45bc1f7e9409a693f7f8ccc56aa91adb5ce6615b7bfb1e4052380dd502163288","name":"penguins.csv","type":"text/csv","lastModified":1760914851121.612,"size":13505},{"id":"2f7252b428e5d1482acf4b4136e75f1704c0dd8dbd77e3edaac6837109d83e33d7777bfba3c2921014ec1310ab7c6fe758c0acedfc7cd1a27eb6c88413748aa4","href":"https://static.observableusercontent.com/files/2f7252b428e5d1482acf4b4136e75f1704c0dd8dbd77e3edaac6837109d83e33d7777bfba3c2921014ec1310ab7c6fe758c0acedfc7cd1a27eb6c88413748aa4","name":"py_loader.png","type":"image/png","lastModified":1760916046199.061,"size":33328}] 2f:{"title":"Framework Guide #5: Data Loaders","cells":[{"id":0,"value":"# Framework Guide #5: Data Loaders\n\n${comment(htl.html`Feel free to refer to this Github Repository, which contains the _Framework_ project that this guide is based off of.`)}","pinned":false,"hidden":false,"mode":"md"},{"id":22,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":37,"value":"## Context: Why a Data Loader?\n\nA **data loader** in the _Framework_ ecosystem is a special script that pre-processes your data to make it easier for you to handle in the actual Markdown files. A data loader is not unique to JavaScript - you can use Python, R, Java, and even Shell commands. This gives you a lot of control over the data that your dashboards use and provides some semantic separation between your front-end code (Markdown files) and your back-end (data-handling processes).","pinned":false,"hidden":false,"mode":"md"},{"id":75,"value":"### Filename Convention\n\nHere's an example of a data loader that's meant to output JSON data:\n\n

src/data/forecast.json.js

\n\nThis file is located inside of the `data/` directory inside `src/`. This is _technically_ a JavaScript file (given the `.js` extension), but when we import the data in our Markdown file, we import it as so:\n\n

const forecast = FileAttachment(\"./data/forecast.json\").json()

\n\nNote that this time, we practically removed the `.js` extension and treated the file as a JSON file, owing to the remaining `.json` extension.","pinned":false,"hidden":false,"mode":"md"},{"id":39,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":52,"value":"When a data loader is found, it will execute it according to which language it interprets the script to be in. This execution **only happens once** and only runs when a new version of your website is built.\n\n${comment(htl.html`Nothing's stopping you from having multiple data loader scripts in your src/data/ directory. When the npm run dev or npm run build commands are run, it will only include data loaders that are actually used. This is pretty similar to to how Observable notebooks, when forked, will not migrate data files that are unused in the original notebook.`)}","pinned":false,"hidden":false,"mode":"md"},{"id":35,"value":"## 1. JavaScript Data Loader\n\nLet's say I want to organize my data and do some data handling with JavaScript before I actually use the data in one of my pages. In this example, I want to load weather data from https://api.weather.gov/. Let's use a JavaScript data loader here.","pinned":false,"hidden":false,"mode":"md"},{"id":70,"value":"### Step 1: Create the loader file\n\nCreate a new folder anywhere in your `src/` directory (_recommendation:_ put it either into a pre-existing `data/` directory or into a new `data/` directory).\n\nInside this directory, we want to create a JavaScript file. The naming convention is important:\n\n

\n# Generic format\n[DATASET_NAME].[EXPECTED FILE EXTENSION].js\n\n# Example:\nforecast.json.js\n

\n\nOur file structure now looks like:\n\n

\n${render_filetree({\n  \"src/\": {\n      \"index.md\": null,\n      \"iris.md\": null,\n      \"quotes.md\": null,\n      \"notebook.md\": null,\n      \"guide.md\": null,\n      \"styling.md\": null,\n      \"data_loaders.md\": null,\n      \"components/\": { \"unique.js\": null, \"quotes.js\": null },\n      \"data\": { \"iris.csv\": null, \"forecast.json.js\": null },\n    },\n})}\n

","pinned":false,"hidden":false,"mode":"md"},{"id":88,"value":"### Step 2: Write your data loader script\n\nYour JavaScript data loader is just like any JavaScript file. Below is the script we've written for `forecast.json.js`:\n\n

\nconst longitude = -117.16;\nconst latitude = 32.72;\n\nasync function json(url) {\n    const response = await fetch(url);\n    if (!response.ok) throw new Error(`fetch failed: ${response.status}`);\n    return await response.json();\n}\n\nconst station = await json(\n    `https://api.weather.gov/points/${latitude},${longitude}`\n);\n\nconst forecast = await json(station.properties.forecastHourly);\n\n// PAY ATTENTION TO THIS LINE:\nprocess.stdout.write(JSON.stringify(forecast));\n

\n\nPlease note that at the end of the script, we use `process.stdout.write()` to \"output\" the JSON data. **This line is absolutely necessary**!","pinned":false,"hidden":false,"mode":"md"},{"id":96,"value":"### Step 3: Import and use your data loader\n\nInside your Markdown file, you can now import your dataset using the `FileAttachment()` syntax. You can then proceed normally from there.\n\nIn our example, this is the way we imported the weather forecast data and formatted the page. To display that the data was properly loaded, we use `Input.table()`.\n\n

\n```js\nconst forecast = FileAttachment('./data/forecast.json').json();\n```\n\n## JavaScript Loader: Forecast Data (Periodic)\n\n### Data Source: \n`https://api.weather.gov/points/[LATITUDE],[LONGITUDE]`\n\n### Preview Coordinates:\n- Latitude: `-117.16`\n- Latitude: `32.72`\n\n```js\ndisplay(Inputs.table(forecast.properties.periods));\n```\n

\n\n${await FileAttachment(\"js_loader.png\").image({width:500})}","pinned":false,"hidden":false,"mode":"md"},{"id":151,"value":"## 2. Python Data Loader\n\nLet's say I'm interested in penguins! I want to record some logistics about them, using the `penguins.csv` file that we attached to this notebook.\n\nWe can us Python to handle our data too. However, unlike JavaScript, there are some important preparations that need to be made.\n\nKeep in mind that _Framework_ expects you to have **Python v3+** installed and accessible via the following shell command:\n\n

\npython3\n

\n\nThis guide will cover how to make sure your environment is set up to properly run Python scripts via _Framework_. Even if you already have Python installed, please read this part of the guide!","pinned":false,"hidden":false,"mode":"md"},{"id":154,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":168,"value":"$34","pinned":false,"hidden":false,"mode":"md"},{"id":170,"value":"$35","pinned":false,"hidden":false,"mode":"md"},{"id":181,"value":"$36","pinned":false,"hidden":false,"mode":"md"},{"id":186,"value":"### Step 5: Import and Use Your Data\n\nNow you can import and use your data similar to how we did [_above_](#cell-96). For our penguin analysis, this is the code we wrote in `data_loaders.md`:\n\n

\n### Data Source:\n<https://github.com/observablehq/data-loader-examples/blob/main/docs/data/penguin-logistic.csv.py>>\n\n```js\nconst penguin_logistics = FileAttachment('./data/penguin_logistics.csv').csv({typed:true});\n```\n\n```js\ndisplay(Inputs.table(penguin_logistics))\n```\n

\n\n${await FileAttachment(\"py_loader.png\").image({width:500})}","pinned":false,"hidden":false,"mode":"md"},{"id":190,"value":"## Quick Reminders\n\n- Remember the naming convention of your data loader files.\n- Use `console.log()` if you want to debug any issues.\n- You must be able to run `python3` in the command line without issue.\n- Always try to run your projects while in a Python environment.\n- Maintain a `requirements.txt` for easy sharing.","pinned":false,"hidden":false,"mode":"md"},{"id":7,"value":"$37","pinned":false,"hidden":false,"mode":"md"},{"id":4,"value":"---\n\n## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":173,"value":"penguins = FileAttachment(\"penguins.csv\").csv({typed:true});","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"import { css, render_filetree, framework_guides, comment } from '@rk2546/2025-infovis-cse_notebook-gallery';","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"css","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W38","stdlib":"1","titling":"auto"}