The nightly CSV drop, turned into a real integration.
The most common enterprise "integration" isn't an API — it's a file that lands in an SFTP folder at 2am. We build and operate the bridge that picks it up, validates every row, and delivers it into your API, reconciled every single run.
"We'll just send you a file every night" is where the easy part ends.
When an enterprise customer can't offer an API, they offer a file: a CSV or fixed-width export, dropped onto an SFTP server (a secure file-transfer folder) by a job someone scheduled years ago. Your sales team says "sure, we can work with that" — correctly, because it is workable. Then the file becomes an engineer's problem.
Parsing a well-formed CSV takes an afternoon. Operating a file-based integration that your customer's business depends on — through format changes, missed nights, half-written uploads, and re-sent files — is an ongoing job. That job is what we take off your team.
The five ways a "simple" file feed fails in production
- Schema drift — someone on their side renames a column, reorders fields, or adds one, and doesn't tell anyone. Without strict validation, wrong data flows silently into your product.
- Partial files — you read the file while it's still being uploaded and ingest half a night's records. The other half never arrives anywhere.
- Encoding surprises — the file claims to be plain text but arrives as Windows-1252, with byte-order marks, embedded commas, or line breaks inside quoted fields.
- Duplicates on re-run — their job fails and re-sends, or overlaps yesterday's records. Without fingerprinting, your API receives the same rows twice and your customer sees double entries.
- Silent missing nights — the export job dies quietly on their side, and nobody notices until the customer asks why last week's data is missing.
Picked up on schedule, validated row by row, reconciled every run.
Pick up and verify the file
Collected on schedule from their SFTP folder, with partial-upload detection so a file still being written is never half-ingested — and an alert if the file simply doesn't arrive.
Validate and transform every row
Encoding normalized, every field checked against a strict schema. A row that fails is quarantined with the exact reason — schema drift gets caught on the next run, not next quarter.
Deliver to your API, exactly once
Every record fingerprinted so re-runs and overlapping files can't create duplicates. Every run must balance: rows in = delivered + skipped + quarantined.
File-feed questions we hear on scoping calls
The file is fixed-width, not CSV. Can you still handle it?
CSV drops are what the bridge runs in production today. The pickup, validation, delivery, and reconciliation pipeline is format-agnostic by design, so a fixed-width or otherwise unusual layout gets scoped as part of the engagement — tell us what the file looks like on the scoping call and we'll give you a straight answer before you commit.
What happens when their team changes the file format without telling anyone?
Validation catches it on the very next run: records that no longer match the schema are quarantined with the exact reason, an alert fires, and we update the mapping. Your customer finds out from us, not from corrupted data in your product.
What if a night's file never arrives, or arrives half-written?
A missing file triggers an alert instead of a silent empty night. Partial uploads are detected before processing starts, so a file still being written is never half-ingested. Both are handled as part of the monthly operate-and-maintain service.
Can the same row ever be delivered to our API twice?
No. Every record is fingerprinted, so re-running a night — or receiving a file that overlaps yesterday's — can never deliver the same record twice. Every run must also reconcile: records in = delivered + skipped + quarantined.
Is a nightly file drop standing between you and a signed deal?
Grab 30 minutes — tell us what the file looks like and what's blocking the close. Pricing works like this →
Prefer email? hello@shapewire.io