Module 1: Ingest anything · Lesson 1.3 · about 35 min
Landing files from folders and shared drives
Land files from shared folders safely: skip half-written files, fingerprint each one with SHA-256, keep a manifest, and never move a source file.
Module 1 · about 35 minutes · read first: 1.2. You’ll be able to land files from a shared folder so that reruns add nothing, re-sends are spotted, corrections are kept as new versions, and half-written files are never read.
1. Why this matters
In many medium-sized companies the most common data source is still a folder: a network share, a SharePoint library synced to a server, a vendor’s export directory. Folder loading looks trivial, and it is where most silent errors live: the file read twice, the file read while it was still being written, the file replaced with a corrected version under the same name.
2. The landing pattern
- Discover: list the candidate files and skip anything not finished.
- Fingerprint: compute a SHA-256 hash of the file’s bytes. A hash is a 64-character code worked out from every byte; change one byte and the code changes completely. Two files with the same hash have the same content.
- Check the manifest: the manifest is your register of every file landed. Is this hash already in it? If so, skip and log
duplicate. - Copy into the raw zone at a path that says the source and the business date:
raw/qsr/business_date=2026-09-14/pos_Q204__a3f91c2e.csv(the first 8 characters of the hash keep names unique). - Record a manifest row: source, original path, size, modified time, hash, when you received it, raw path.
You copy, you never move. The source folder belongs to someone else. Moving files breaks their process and destroys your ability to check the source again.
import hashlib, shutil, sqlite3, time
from pathlib import Path
def sha256(path: Path, chunk=1 << 20) -> str:
h = hashlib.sha256()
with path.open("rb") as f:
for block in iter(lambda: f.read(chunk), b""): # 1 MB at a time: works for huge files
h.update(block)
return h.hexdigest()
def is_stable(path: Path, wait=2.0) -> bool:
"""A file still being written keeps growing. Skip partials and temp names."""
if path.suffix in {".partial", ".tmp", ".crdownload"} or path.name.startswith("~$"):
return False
s1 = path.stat().st_size; time.sleep(wait); s2 = path.stat().st_size
return s1 == s2 and s1 > 0
def land(src_dir: Path, raw_dir: Path, db: sqlite3.Connection, source: str):
db.execute("""create table if not exists manifest(
sha256 text primary key, source text, src_path text, size int, mtime real,
received_at text, raw_path text)""")
for p in sorted(src_dir.rglob("*.csv")):
if not is_stable(p):
continue
digest = sha256(p)
if db.execute("select 1 from manifest where sha256=?", (digest,)).fetchone():
continue # exact same bytes already landed
business_date = p.parent.name # in this source, the folder is the date
dest = raw_dir / source / f"business_date={business_date}" / f"{p.stem}__{digest[:8]}{p.suffix}"
dest.parent.mkdir(parents=True, exist_ok=True)
tmp = dest.with_suffix(dest.suffix + ".tmp")
shutil.copy2(p, tmp); tmp.replace(dest) # rename is atomic: no half files
db.execute("insert into manifest values (?,?,?,?,?,datetime('now'),?)",
(digest, source, str(p), p.stat().st_size, p.stat().st_mtime, str(dest)))
db.commit()
Watch one run over a folder with four files in it:
pos_Q207.csv.partial is still being uploaded, so it is held back until it is renamed.pos_Q204.csv hashes to a3f91c2e…, which is not in the manifest. It is copied to raw and a row is added.pos_Q205.csv hashes to 7be04d19…, already in the manifest from yesterday. Same bytes, so it is skipped and logged as a duplicate.pos_Q206.csv has yesterday's name but a new hash: someone fixed the file. It lands as a second version; staging will use the latest.Maths and principle
Principle: a step is idempotent if doing it twice has the same effect as doing it once. Landing keyed by content hash is idempotent.
Equation: . With a manifest (a set of hashes) and a file with hash :
because a set ignores repeats.
Analogy: pressing a lift button twice; the lift still comes once. Where it breaks: the same business data can arrive with different bytes (a re-export with a new “printed at” line). Then the hash differs, and you need a second, business-level key (source, store, business date) with a rule such as “latest version wins”.
Try it: a folder has 18 files today; 2 are byte-identical re-sends of yesterday’s. How many manifest rows does today add if the job runs twice? (16. The re-sends and the second run add nothing.)
3. Same name, new content
Vendors often fix a file and drop it again under the same name. Decide the rule out loud and write it in the README:
| Situation | Hash | Name | Rule |
|---|---|---|---|
| Exact re-send | same | any | skip |
| Correction | new | same | land as a new version; staging takes the latest by received time |
| New day | new | new | land |
Never trust the file’s modified time alone: copy tools, sync clients and unzipping all rewrite it. Never trust dates in file names alone either: people rename files. Record name, modified time and received time, and choose one business date rule per source.
- Hash (SHA-256)
- A fingerprint of a file's bytes. Same bytes, same fingerprint; one byte different, a completely different one.
- Manifest
- Your register of every file landed: where it came from, its hash, when it arrived and where the copy lives.
- Idempotent
- Safe to repeat: running it again changes nothing.
- Atomic rename
- A rename that happens all at once, so nobody can see a half-written file.
- Physical meaning
- Six months from now, finance will ask why March's numbers changed. The manifest answers in one query: which version of which file fed which load, and when it arrived. Because the job is idempotent, you can also rerun it after any failure without fear of double-counting.
“If the file name is in the manifest, I already have it.” Names lie: a corrected file can reuse yesterday’s name, and two different stores can send files with the same name. Key the manifest on the hash of the content, and keep the name only as information.
4. Service accounts, mapped drives and polling
A scheduled job usually runs as a service account (a login for a program, not a person). The Z: drive you mapped in your own session does not exist for it. Use full network paths (\\fileserver\exports\qsr) and give the service account read-only access. Read-only is a feature: it makes deleting the source impossible.
You can poll (list the folder every few minutes) or watch (ask the operating system for change events, for example with Python’s watchdog). Polling is simpler and works on network shares, where events are unreliable. It is what Dagster sensors do in Module 3.
5. Try it: land the restaurant shared drive
- In
retail-ingest, createsrc/retail_ingest/sources/folder.pywithsha256,is_stableandland(usewait=0.2locally). - Run it over
data/landing/shared_drive/qsr. Count the manifest rows; it should equal the number of files. - Run it again: zero new rows. Write
tests/test_folder_idempotent.pythat proves it, using pytest’stmp_path. - Simulate a correction: add a line to one source file and rerun. Check a second version lands with a new hash.
- Simulate an upload in progress: create
pos_Q999.csv.partialand check it is skipped. - Move the manifest to DuckDB (
duckdb.connect("data/manifest.duckdb")) and query files per day, total bytes, and any day with fewer than 18 files.
Commit: feat(ingest): idempotent folder landing with sha256 manifest.
What to check: both tests pass, and your “fewer than 18 files” query returns nothing for the restaurant source (each of the 18 restaurants sends one file a day).
6. Practice
- A folder holds 18 files. You run the job three times. How many manifest rows?
- Why does
is_stablecheck the size twice? - The sync client rewrote every file’s modified time overnight. Does your job re-land them?
Answers
-
- The second and third runs add nothing.
- A file still being written grows between the two checks. Same size twice (and above zero) suggests the writer has finished.
- No. The hashes have not changed, so the manifest already has them.
New words: hash, SHA-256, manifest, idempotent, atomic rename, service account, polling.
- Copy, never move: the source folder belongs to someone else.
- Fingerprint every file with SHA-256 and keep a manifest keyed on that hash.
- Skip half-written files (temp names, growing sizes) and write your own copies to a temp name, then rename.
- A correction under the same name has a new hash: land it as a new version and let staging take the latest.
Check yourself
Sources
- Python documentation: hashlib (secure hashes) https://docs.python.org/3/library/hashlib.html
- Python documentation: pathlib and os.replace https://docs.python.org/3/library/pathlib.html
- Kleppmann, M. (2017), Designing Data-Intensive Applications, O'Reilly Media, Chapter 11 (idempotence and exactly-once effects)
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.