What is built, in full

Every mode, transport and record PIGEON produces, with the limit each one carries stated in the same breath as the capability. One portable folder holding a graphical app and a command-line tool: no installer, nothing to register, nothing to phone home to.

Ways to move data

Mode	What it does	Best for
Direct copy	Multi-threaded copy with resume. An interrupted run picks up where it stopped instead of starting over, and a file that failed verification is deliberately not checkpointed, so the resume re-copies it.	Local disks, shares, removable media
Archive and go	Packs the source into one compressed stream, unpacks it at the destination, then hashes every file that landed. A single stream has no halfway point, so an interrupted run restarts rather than resumes — and the failure state reported for it is “discarded”, not “partial”.	Huge numbers of small files, slow links
Differential sync	Compares the source against what the destination already holds, by content hash, and sends only the files that differ. Not size, not timestamps, not the two together — so an edit landing in the same second as the previous run is still seen. Mirror-delete is available and off by default.	Repeat runs to the same destination
Offline batch	Packs only what changed into one batch file, signed with your own key, for hand-carrying across an air gap. The far end checks the signature and its own starting state before writing anything, and verifies the whole result afterward.	Disconnected systems that need regular updates

Every mode re-reads what landed at the destination and hashes it there, never the copy of it that went past on the way. Verification is on by default in all eleven profiles that ship.

Reaching the far end

A transfer can run entirely against paths the machine already has mounted. It can also cross a network — PIGEON either connects out, or waits for something to connect to it. Every network role is off until a profile or a command turns it on.

PIGEON connects out

SFTP, through the host’s own OpenSSH client
HTTPS, with mutual-TLS client certificates, through the host’s own curl
FTPS with explicit TLS, through the same client

PIGEON bundles no protocol implementation of its own for any of these. It drives the tools the host already ships and trusts, so your hardened SSH configuration is the boundary rather than a vendor’s black box. No credential is ever written to a profile or to disk: a profile holds a host, a path, and at most the path to an identity file. Plain unencrypted FTP is not implemented at all, in either direction. TFTP exists as a gated option for equipment that speaks nothing else — left out of builds by default, and refused at runtime even in a build that has it unless a separately signed authorisation is present, verified against a trust root kept deliberately separate from the one every other PIGEON artifact is checked against.

PIGEON is the far end

serve-sftp — the host’s own OpenSSH server, public-key only
serve-https — an embedded server, mutual TLS, client certificates you add by hand
serve-watch — not a listener. It watches a drop directory and applies each batch that lands there, setting aside anything it refuses rather than deleting it

All three are proven end to end on Linux, and serve-sftp is proven on real Windows too: the server brought up, a real client completed a key exchange against it, and an upload landed in the served directory, which is what proves it served the right folder rather than merely accepting a connection. The embedded HTTPS server’s suite runs as a native Windows binary in continuous integration. Two limits travel with these: Windows OpenSSH has no chroot directive at all, so on Windows serve-sftp runs unconfined — still restricted to SFTP, but confinement is not claimed there; and the embedded server’s trust bundle is read once when the listener starts, so removing a client certificate takes effect at the next restart rather than immediately. Both are documented behaviour rather than things to discover.

Differential sync and resume work over all four remote transports. Archive-and-go and the offline batch workflow deliberately do not: both need something at the far end to act on what arrives, and none of SFTP, HTTPS, FTPS or TFTP has that. Adding it would mean designing a new wire protocol rather than filling in a missing code path, so it is a stated scope line.

Inside the offline batch workflow

This is the air-gap half of the product, and most of the engineering. A batch is one signed file a courier carries; everything here exists either to make that file smaller or to make it impossible to misapply.

Smaller

File-level differencing against a small local record of what the far end is believed to hold, so unchanged files never travel. Block-level deltas for a file whose previous version the far end still has, from any of three sources: a live readable copy of the destination, a signature carried back on a courier’s return leg, or a local basis cache the workflow maintains for itself. Content-addressed chunking for the case deltas cannot reach: five machine images sharing one base layer carry that layer once, across files and across runs, rather than once per image.

Every one of those is checked before it is used. A candidate basis is re-hashed and compared against what the far end’s last recorded state says for that path; a mismatch falls through to sending the whole file rather than producing a delta against content the far end no longer has. A delta that does not come out at least ten per cent smaller than the file is discarded. And the far end re-verifies the reconstructed file against the hash the sender promised before it replaces anything, into a temporary file inside the destination directory so the final replace is an atomic same-filesystem rename.

Harder to misapply

Base-state checking: an incremental batch names the state it was built against, and the far end refuses it if that is not what the far end actually holds. Sequence chaining for a one-way stream, where the base check cannot help because a one-way snapshot has no base: each batch carries a session, a sequence number and the hash of the batch before it, so a replayed or rolled-back delivery is refused and a gap is recorded rather than silently accepted. Media identity: the batch can record the identity of the volume it was written to, and an apply from a different volume is recorded as a custody event rather than blocked. Encryption to a recipient key for a relay you do not trust, with the file names and paths inside the encrypted payload rather than the cleartext header.

A batch header used to embed the complete far-end file list, which gave a batch a size floor proportional to the file count at the far end rather than to the change: around thirteen megabytes of header for a hundred-thousand-file tree, on a one-byte edit. It now carries only what changed and the far end reconstructs the rest — but only when the base state provably matches, because the naive version of that optimisation would quietly accept whatever happened to be sitting at the destination as vouched for. When the base does not match, the reconstruction is skipped and the result is flagged as having incomplete per-file diagnostics rather than silently wrong ones.

Running unattended

A transfer that has to survive a logoff or a power loss runs through a background service: a real Windows service registered with the service manager, or a systemd unit. Jobs are submitted to a persistent queue, one file per job, written atomically.

The worker never moves a byte of its own. It looks the profile up at execution time and calls the same transfer engine everything else calls, so verification, resume, differential and checkpointing apply automatically and there is no second, weaker path for data.
A job is marked running, and its attempt counted, before the transfer starts. If the process dies mid-transfer the on-disk record already says so, which is what crash recovery needs. Recording the attempt afterward would make a crash indistinguishable from a job never attempted.
Crash recovery never reads an interrupted job as either succeeded or failed — only as unknown, and requeues it if it has attempts left. How much of the file needs re-sending is not something the queue tracks; that comes from the resume and checkpoint machinery already in the engine.
A stop is not a failure. A service stop cancels the transfer in flight, returns the job to the queue and refunds the attempt.
Priority ordering across the queue, with ties broken by submission time, and retry with exponential backoff up to a cap. A job naming a profile that does not exist fails outright rather than burning through its retry budget on something that will not fix itself.
Every transition is written to the same hash-chained audit log as everything else, not only to the job file, so an unattended run is as reviewable as one someone watched.

Profiles for common jobs

A profile holds the settings for a kind of transfer: how it moves, how many threads, compression, bandwidth cap, and which files to include or skip. Eleven ship with the product, covering the shapes below. They are plain files: edit one, write your own, or share one with another site. A profile you write that carries a built-in’s name takes precedence over it, and deleting your file restores the original. A profile file that will not parse is refused by name rather than silently replaced by the built-in that shares its id — substituting a different mode, a different destination shape and a different deletion policy under the same name is worse than stopping.

The job	What the profile sets up
Deep, intricate trees of geospatial or environmental product data — a hundred thousand files and very long paths — over a long-haul, constrained link	Archive-and-go, compression, four threads, verification, resume, and exclusions for temporary and cache folders
An extremely narrow link, the kind measured in tens of kilobits	Differential batches, maximum compression, metadata stripping, a hard bandwidth cap, single-threaded
Very large sequential files — machine images, server backups — over a fast local network	Light compression, eight threads, full verification, resume
Enormous numbers of small files, such as an image or product library	Archive-and-go, light compression, six threads, and exclusions for the thumbnail and metadata files such libraries collect
Ordinary bulk movement inside a trusted network	Light compression, six threads, verification, resume
An older or more constrained destination that needs short, plain filenames	Archive-and-go, light compression, two threads, conservative name safety
A steady one-way flow across a diode, to a PIGEON receiver running unattended	Offline batches, differential, one-way, and replay chaining. Every batch is a complete snapshot; the receiver checks each one and refuses a replay
A single one-way delivery on removable media	The same one-way posture, plus a record of the volume the batch was written to. The far end sets aside anything the snapshot does not list
A relay you do not fully trust — a shared drop, a cloud folder, a courier you cannot vouch for	One-way batches with replay chaining and media identity, meant to be paired with the receiver’s public key so the batch is sealed to them
A transfer that has to survive a logoff or a power loss	Runs through the background service’s job queue: verified, resumable per file, retried with backoff until it lands
Everything protective on, running in front of you	Verification, resume, per-file byte-range resume, sparse-file aware, no background service. Deliberately no deletion and no move, so there is nothing set aside that it could mislead you about

What’s included

Delivery and integrity

SHA-256 receipt for every run, built from the files that landed — never from the copy of them that went past on the way
Per-file status in that receipt: delivered, missing, mismatched or skipped, rather than one overall verdict
Degradation notices recorded in the receipt itself, so a run whose profile asked for something it did not get says so on the delivery record rather than in a log line that scrolls past
Change detection by content hash, with an optional timestamp cache that carries two independent guards and a periodic full rescan for the edit no timestamp rule can see
Resume after interruption, and per-file byte-range resume over SFTP both ways, FTPS both ways, and HTTPS pull
Batches you sign with your own key; the far side verifies the signature, and can be set to refuse an unsigned batch
Package self-check, to confirm the folder in front of you is exactly what was issued

Operation

A readiness check that runs before a byte moves and reports every problem it finds at once, rather than one per attempt — among them the source, the destination, free space, the tools each transport needs, long-path support, the audit log’s own writability, the clock, and — for a remote destination — whether the far end is reachable and has room
Plain-language errors with stable codes, a category, a remedy, and an explicit answer about whether the destination is usable
Exit codes a script can act on, mapped one-to-one from the error category and held that way by a test
Bandwidth cap shared as one budget across all worker threads rather than per thread, on a schedule if you want one, with a burst allowance capped at about one second of the configured rate so an idle limiter cannot bank credit and then blast
Long paths beyond the Windows 260-character limit handled natively throughout, including the directory walk
A dual-pane file explorer for browsing a local tree beside a remote one, which pushes and pulls through the same engine as everything else — there is no weaker path for “just browsing”

Records

Hash-chained audit log of every run, with one chain shared across processes, so a watcher daemon and an interactive run do not each keep their own idea of the tail
Each receipt carries a reference to the audit record that opened its run, so a log truncated at the end is caught — a hash chain alone cannot detect that
A read-only history view in the app, built by reading that same log rather than by keeping a second record, so the graphical and command-line views cannot drift apart
Optional projection of the audit log to a SIEM over syslog, CEF or LEEF, carrying each record’s sequence number, hash and previous-hash so a collector can detect a gap the same way the local verifier does. Off unless an operator configures a collector; a collector outage loses nothing, because the local file stays authoritative and can be replayed from any point
Software bill of materials produced at packaging time, read from what the binaries actually link — including statically linked components, which an import table alone cannot see — and signed

Signing and trust

Signatures over PIGEON’s own artifacts are hybrid: a classical one, ECDSA P-384 with SHA-384, and a post-quantum one, ML-DSA-87. Verification requires both, so breaking either algorithm alone forges nothing
The set of algorithms present is bound into what each signature covers, so the post-quantum half cannot be stripped off and the classical half presented as a legitimate classical-only signature. Both the stripped form and the same bytes rewritten as an older-format file are refused
Two independent signing systems, deliberately not interchangeable: one for PIGEON’s own artifacts against a trust root compiled into the build, and an ordinary code-signing certificate for the Windows executables, which is a different trust mechanism the operating system understands natively
An optional content-authorization gate: every file that would travel must carry a detached signature from a key in a trust bundle a technician placed by hand, checked before a byte moves, with the whole run refused if any file lacks one. It verifies that a release decision was signed; it does not make one
An integration seam for an external, accredited content-inspection tool, run over every file before the transfer starts. PIGEON inspects no content itself, and a tool that modifies a file it was asked only to check is detected and the run refused
No update checks, no crash reporting, no telemetry. PIGEON only reaches out to the host your own profile names, and only listens when you start one of the server modes yourself

Where it runs

Platform	Command-line tool	Graphical app
Windows 10, 11, Server 2016 and later	Supported	Supported
Modern Linux	Supported	Supported

Both platforms are built and tested in continuous integration on their own hardware — Windows natively, with a Windows-hosted compiler, rather than cross-built and emulated.

The legacy far end

A separate, much smaller tool exists for a destination the main product will never run on. It is written to a 2003-era platform floor, shares no code with the main codebase — not even its JSON reader, because sharing one would drag the main product’s language requirements onto a target that exists to escape them — and imports only the two core Windows system libraries. It can extract a batch, verify what landed against the manifest, and check a signature.

Three things travel with that, because they decide whether it fits your situation. It is receive-only: a legacy far end can be delivered to and cannot originate a transfer. Its signature check verifies the classical half of a hybrid signature and says plainly, in its own output, that it did not check the post-quantum half — a full check needs the main tool. And while it builds and is tested in continuous integration, it is unproven on real period hardware, which no amount of continuous integration settles. If that is your far end, tell us what it is and we will tell you what we actually know.

What PIGEON doesn’t claim

PIGEON does not grant an authorization to operate. It gives your assessors clear records, signatures and documentation to work with.

PIGEON is not a content filter or sanitiser, and the hook that runs one is an integration seam for an accredited tool rather than a filter of PIGEON’s own. The content-authorization gate checks that a release decision was signed; it is not a release authority.

Package signing and the audit trail are tamper-evident, not tamper-proof: they tell you a record was altered, removed or reordered. They do not stop someone with write access replacing a file wholesale. Host controls such as application allow-listing still matter.

Naming an algorithm is not a validation claim. The algorithms PIGEON signs with are published standards; that is a property of the algorithm, and says nothing about whether any particular implementation of it has been independently validated. PIGEON is not itself a validated cryptographic module and does not claim to be.

Verification is an application-layer guarantee. It detects a corrupted delivery that a one-way device’s own link-layer error correction failed to correct; it does not replace that error correction, and over a true one-way link it cannot correct anything.

The full set of scope lines, including what each mode will not do and why.

A specific question beats a brochure

Tell us what the data is, what it has to cross, and what runs at the far end. We will tell you plainly whether PIGEON fits — including when it does not.

Ask a question