TechByteByByte

Files and Data

Learn how to read and write text files, CSV data, and JSON in Python, and how these map directly to the request and response format used by every AI API.

#Python#Files#JSON#CSV#Data Handling#AI#Python for AI

The problem: Values in memory normally disappear when Python stops, but AI applications need documents, datasets, settings, logs, cached embeddings, and evaluation results to survive. They must also exchange structured data, commonly as JSON.

What you will learn: You will follow data between Python objects and stored bytes using safe file handling, paths, UTF-8 text, CSV, JSON, and JSONL. You will also separate three jobs that are often confused: reading bytes, parsing a format, and validating what the data means.

1. File Handling

A Python variable lives in memory and normally disappears when the process ends. A file stores bytes on a device so another runโ€”or another programโ€”can use the information later.

Python object in memory
      โ†“ encode or serialize
bytes stored in a file
      โ†“ read, decode, or parse
new Python object in memory

Plain text needs an encoding such as UTF-8. JSON and CSV add rules for representing structured values. Reading a file does not prove its content is valid or safe, so parsing and validation are separate jobs.

Memory That Survives the Program

Reading and writing files lets your program persist data beyond a single run โ€” and load in data (documents, datasets) that already exists.

Picture a File as Stored Bytes

A file is a notebook page: you can open it, read whatโ€™s written, add new lines, or start a fresh page.

Why Programs Need Persistent Data

Variables disappear the moment your program ends. Files let data survive between runs โ€” a document you want to process, a log you want to review later, results you want to save.

2. Reading Files

Syntax

with open("filename.txt", "r") as f:
    content = f.read()

Example

# Assume notes.txt already exists with some text in it
with open("notes.txt", "w") as f:
    f.write("Python is a great language for building AI applications.\n")
    f.write("Retrieval-Augmented Generation combines search with generation.\n")

with open("notes.txt", "r") as f:
    content = f.read()

print(content)

Expected Output:

Python is a great language for building AI applications.
Retrieval-Augmented Generation combines search with generation.

How It Works

  • open(filename, mode) opens a file. "r" = read, "w" = write (overwrites), "a" = append.
  • with ... as f: is a context manager (youโ€™ll learn the general concept in Module 10) โ€” it automatically closes the file when youโ€™re done, even if an error occurs. Always prefer this over manually calling f.close().
  • f.read() returns the entire fileโ€™s contents as one string.

โš ๏ธ Common Beginner Mistake: Forgetting with and manually managing open()/close() โ€” if an error happens between them, the file never gets closed properly, which can corrupt data or leak file handles. Always use with.

Reading line by line

with open("notes.txt", "r") as f:
    for line in f:
        print("Line:", line.strip())

Expected Output:

Line: Python is a great language for building AI applications.
Line: Retrieval-Augmented Generation combines search with generation.

๐Ÿค– How Is This Used in AI? Loading a document into memory before cleaning, chunking, and embedding it โ€” the very first step of a RAG ingestion pipeline.

3. Writing Files

summary = "Python is essential for AI development."

with open("summary.txt", "w") as f:
    f.write(summary)

๐Ÿค– How Is This Used in AI? Saving a modelโ€™s generated output, saving processed/cleaned text before embedding, or writing evaluation results to disk for later review.

4. Appending Files

with open("log.txt", "a") as f:
    f.write("Request processed successfully.\n")

๐Ÿง  Intuition: "w" starts a fresh page every time (erasing what was there); "a" keeps adding to the bottom of the existing page.

๐Ÿค– How Is This Used in AI? Logging every request an AI service handles โ€” you want each new log entry added, never overwriting the ones before it.

โš ๏ธ Common Beginner Mistake: Using "w" when you meant "a" โ€” silently erasing an entire log file every time your program runs is a very common, very avoidable bug.

5. Working with Paths

from pathlib import Path

data_folder = Path("data")
file_path = data_folder / "documents" / "notes.txt"

print(file_path)
print(file_path.name)
print(file_path.suffix)

Expected Output:

data/documents/notes.txt
notes.txt
.txt

๐Ÿง  Intuition: pathlib.Path treats file paths as objects you can combine with /, instead of fragile string concatenation that behaves differently on Windows vs. Mac/Linux.

๐Ÿค– How Is This Used in AI? Locating a folder of documents to ingest, or building the output path for a saved embeddings file โ€” pathlib is the modern, reliable standard for this in real projects.

6. CSV Files

What Is It?

CSV (Comma-Separated Values) is a simple, common format for tabular data โ€” rows and columns, like a spreadsheet, stored as plain text.

import csv

# Writing a CSV of evaluation results
rows = [
    {"question": "What is Python?", "score": 0.9},
    {"question": "What is RAG?", "score": 0.85},
]

with open("results.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["question", "score"])
    writer.writeheader()
    writer.writerows(rows)

# Reading it back
with open("results.csv", "r") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row)

Expected Output:

{'question': 'What is Python?', 'score': '0.9'}
{'question': 'What is RAG?', 'score': '0.85'}

โš ๏ธ Note: Everything read from a CSV arrives as a string โ€” notice 'score': '0.9' has quotes. Youโ€™d need float(row["score"]) to use it as an actual number (recall Module 1โ€™s type conversion).

๐Ÿค– How Is This Used in AI? Storing model evaluation results, labeled training examples, or benchmark datasets โ€” CSV is still extremely common for tabular AI data, especially before it gets loaded into Pandas (Module 9).

7. JSON

What Is It?

JSON (JavaScript Object Notation) is a text format for structured data built from the exact same shapes as Python dicts and lists โ€” which is why it maps so cleanly onto Python.

{
  "model": "gpt-4o-mini",
  "temperature": 0.7,
  "messages": [{ "role": "user", "content": "Hello" }]
}

Python converts dictionaries and lists to and from JSON using the standard library json module:

graph LR
    subgraph Python Memory
        dict["Python Dict / List"]
    end
    subgraph Disk or Network API
        jsonStr["JSON String (raw text)"]
    end
    dict -->|"json.dumps() / json.dump()"| jsonStr
    jsonStr -->|"json.loads() / json.load()"| dict

๐Ÿง  Intuition

JSON is the universal shipping format for structured data across the internet โ€” nearly every API (including every LLM API) sends and receives JSON.

8. Reading JSON

import json

json_text = '{"model": "gpt-4o-mini", "temperature": 0.7, "messages": [{"role": "user", "content": "Hello"}]}'

data = json.loads(json_text)   # JSON string -> Python dict
print(data["model"])
print(data["messages"][0]["content"])
print(type(data))

Expected Output:

gpt-4o-mini
Hello
<class 'dict'>

Reading from a .json file works the same way, using json.load(f) (no โ€œsโ€) on an open file object:

with open("config.json", "r") as f:
    config = json.load(f)

[!IMPORTANT] Production Alert: Dealing with Markdown JSON Fences from LLMs In production RAG and agent architectures, you will often ask an LLM to generate output formatted strictly as JSON. However, LLMs have a strong habit of wrapping their JSON outputs inside markdown code blocks:

```json
{
  "action": "search",
  "query": "vector databases"
}
```

If you feed this raw text directly to json.loads(), Python will crash with a json.JSONDecodeError because standard JSON parsers do not recognize markdown fences (```json and ```) as valid JSON.

To handle this defensively in production, you should strip the markdown fences before calling loads():

raw_response = """```json
{"action": "search", "query": "vector databases"}
```"""

# Clean up markdown fences if present
clean_json_text = raw_response.strip()
if clean_json_text.startswith("```json"):
    clean_json_text = clean_json_text[7:]
if clean_json_text.endswith("```"):
    clean_json_text = clean_json_text[:-3]
clean_json_text = clean_json_text.strip()

data = json.loads(clean_json_text)
print(data["query"]) # 'vector databases'

9. Writing JSON

result = {
    "question": "What is RAG?",
    "answer": "RAG combines retrieval with generation.",
    "sources": ["doc1.txt", "doc2.txt"],
    "confidence": 0.87
}

json_string = json.dumps(result, indent=2)   # Python dict -> JSON string
print(json_string)

with open("result.json", "w") as f:
    json.dump(result, f, indent=2)   # write directly to a file

Expected Output:

{
  "question": "What is RAG?",
  "answer": "RAG combines retrieval with generation.",
  "sources": [
    "doc1.txt",
    "doc2.txt"
  ],
  "confidence": 0.87
}

๐Ÿง  Intuition: loads/dumps work with strings; load/dump (no โ€œsโ€) work directly with open files. indent=2 just makes the output human-readable.

๐Ÿ’ก JSONLines (JSONL) for AI Datasets & Fine-Tuning

When preparing datasets for fine-tuning (like OpenAI fine-tuning datasets) or when saving massive streams of logs, you will use JSONLines (.jsonl) instead of a single massive JSON array.

In a JSONLines file, every single line is a self-contained, valid JSON object, separated by a newline character (\n). This allows you to stream and read files line-by-line without loading a massive 10GB JSON array into RAM at once.

Writing a .jsonl file:

import json

dataset = [
    {"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}]},
    {"messages": [{"role": "user", "content": "Solve 2+2"}, {"role": "assistant", "content": "4"}]},
]

with open("dataset.jsonl", "w") as f:
    for item in dataset:
        f.write(json.dumps(item) + "\n")

Reading a .jsonl file:

with open("dataset.jsonl", "r") as f:
    for line in f:
        item = json.loads(line.strip())
        print(item["messages"][0]["content"])

10. JSON in AI APIs

This is the payoff โ€” every LLM API call is built from exactly what you just learned:

import json

# What you SEND to an LLM API is built as a Python dict, then becomes JSON
request_payload = {
    "model": "gpt-4o-mini",
    "temperature": 0.7,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is retrieval-augmented generation?"}
    ]
}

request_json = json.dumps(request_payload)
print(request_json[:80], "...")

# What you RECEIVE back is JSON text, parsed into a Python dict
response_json = '''
{
  "id": "msg_01",
  "model": "gpt-4o-mini",
  "content": [{"type": "text", "text": "RAG combines search with generation."}],
  "usage": {"input_tokens": 20, "output_tokens": 8}
}
'''
response = json.loads(response_json)
answer = response["content"][0]["text"]
print(answer)

Expected Output:

{"model": "gpt-4o-mini", "temperature": 0.7, "messages": [{"role" ...
RAG combines search with generation.

๐Ÿค– This is what happens, almost line for line, inside every AI SDK you will ever call: build a dict โ†’ convert to JSON โ†’ send over HTTP โ†’ receive JSON back โ†’ parse into a dict โ†’ pull out the fields you need.

11. Processing Structured Data

Combining files + JSON โ€” a realistic small pipeline:

import json

documents = [
    {"text": "Python is used heavily in AI development.", "source": "notes.txt"},
    {"text": "RAG improves LLM answers with retrieved context.", "source": "notes.txt"},
]

# Save processed documents as a JSON dataset, ready for embedding later
with open("processed_docs.json", "w") as f:
    json.dump(documents, f, indent=2)

# Load them back for the next pipeline stage
with open("processed_docs.json", "r") as f:
    loaded_docs = json.load(f)

for doc in loaded_docs:
    print(f"[{doc['source']}] {doc['text']}")

Expected Output:

[notes.txt] Python is used heavily in AI development.
[notes.txt] RAG improves LLM answers with retrieved context.

๐Ÿค– Saving intermediate results as JSON between pipeline stages (ingestion โ†’ cleaning โ†’ chunking โ†’ embedding) is standard practice โ€” it lets you inspect, debug, or resume a pipeline without redoing expensive earlier steps.

Text Encoding: How Characters Become Bytes

A text file ultimately stores numbers called bytes. An encoding is the rule that maps those bytes to characters. UTF-8 can represent English, Hindi, emoji, and many other writing systems, so state it explicitly when practical:

from pathlib import Path

path = Path("notes.txt")
path.write_text("Hello เคจเคฎเคธเฅเคคเฅ‡ ๐Ÿ‘‹", encoding="utf-8")
text = path.read_text(encoding="utf-8")

Using the wrong encoding can produce strange characters or an error. For a very large file, process one line at a time instead of loading the entire file into memory.

Safe Writes and Data Formats

If a program crashes halfway through overwriting an important file, the file may be incomplete. Production systems commonly write the complete new content to a temporary file and then replace the destination in one final operation.

Use JSON for one structured object or a manageable array. Use JSONL when each line is an independent record; a training pipeline can then stream one example at a time and can often recover more easily from one malformed line.

Module Summary

You can now read and write text files safely with with open(...), understand file modes (r/w/a), build reliable paths with pathlib, read and write CSV tabular data, and โ€” most importantly โ€” convert between JSON text and Python dicts/lists in both directions.

AI Connection

JSON is the shared language between your Python code and every AI API. Every request you send and every response you receive is JSON underneath, translated to and from Python dicts using exactly json.dumps/json.loads you just practiced. Files are how your pipeline remembers documents, results, and intermediate data between runs.

Mini Practice

  1. Write a documentโ€™s text to a file, then read it back and print its word count.
  2. Create a Python dict representing an LLM API request (model, messages), convert it to a JSON string with json.dumps, and print it.
  3. Parse a JSON string representing an API response and extract just the answer text and token usage.
  4. Write a list of 3 dictionaries (each with text and score) to a JSON file, then load it back and print only the ones with score >= 0.7.
  5. Explain, in one or two sentences, why nearly every AI API โ€œspeaks JSON,โ€ and how that connects to what you learned about dictionaries in Module 2.

Next: Module 8 โ€” Modules, Packages and Environments โ€” organizing code and securely managing API keys.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed