documentation/S3.md · main · 2026-08-28

S3 Remote Backend

Any S3-compatible bucket (AWS S3, Cloudflare R2, DigitalOcean Spaces, MinIO, etc.) can be a git remote. The backend does two things:

  1. Git remote — a git remote helper for s3:// URLs (per GITMSG.md §1.3), implemented in core/objstore, so a bucket works as a normal push/pull remote. This doc owns the transport.
  2. Static site — an optional read-only website uploaded alongside a bucket-hosted repo, so the repo is browsable without git. See STATIC-SITE.md.

Quick start

gitsocial clone s3://<endpoint>/<bucket>/<prefix>   # clone from a bucket
gitsocial remote add <url> --default --site         # register a bucket as the default push target, site on
gitsocial push                                      # publish repo data (+ the website once site.publish is on)

clone and remote add both accept a pasted AWS S3 console URL and normalize it. remote add --default appends the remote to the multi-valued gitsocial.pushRemote defaults; --site enables site.publish for the repo. You can also add an s3:// URL as a git remote by hand and push/pull normally.

Push behavior

gitsocial push [remote] publishes the repository: gitmsg data branches, state refs, tags, reason-based code branches (the default branch when ahead, plus open-PR heads), and — for s3:// remotes with the site.publish guard enabled — the browsable static site alongside the data. Flags only say no or expand scope:

Non-fast-forward on an --all-branches branch is reported per-branch (never auto-merged); reconcile with an explicit git push --force-with-lease.

Output names the remote (Pushed to <remote> (<url>)) and adds a Site: published / Site: skipped (<reason>) / Site: failed: <err> line. A site failure after a successful data push is a warning, not an error: the data push stands.

Multiple remotes. gitsocial push [remote...] accepts several remotes (gitsocial push r2 s3local) and pushes them sequentially, one result block each: a failed remote is reported and skipped rather than aborting the run, and the command exits non-zero if any remote failed. With no positional argument the (multi-valued) gitsocial.pushRemote default is used, so a bare gitsocial push can fan out to every configured default. Counting, tags, auto-merge, and site maintenance run per remote exactly as for a single one.

Tracking-ref reconcile (s3:// only). Before a real push (not --dry-run, which stays fully offline), gitsocial lists the bucket's refs/ once and syncs the remote's tracking refs to it — deleting tracking refs whose bucket ref is gone and updating ones that differ. Non-s3:// remotes are unchanged.

Site gate. site.publish=true is the only enabler, and it lives in the pushed config ref (refs/gitmsg/core/config), so it travels with the repo: both gitsocial push and the git remote helper's own post-push maintenance (a plain git push s3://…) honor it, creating the site on a fresh bucket and maintaining an existing one. Without it every push moves repo data only — a bucket that already carries a site is left untouched with a one-line hint naming the config to set.

Push remote resolution

gitsocial push [remote...] (with or without --site-only) resolves the target remotes in this order: the positional [remote...] arguments, then git config gitsocial.pushRemote, then a heuristic (origin if it is an s3:// remote, else the first s3:// remote alphabetically, else origin). The gitsocial.pushRemote key is multi-valued: set one or several per-clone defaults with gitsocial remote default <name...>, and run it with no argument to print the current resolution. When the heuristic must choose among two or more s3:// remotes with nothing configured, it warns once on stderr naming the remote it picked and the config to set. A configured remote that no longer exists is dropped from the set (falling through to the heuristic with a stderr warning if none remain).

Per-remote site overrides. A remote can override the site's deployment keys — url, publish, pages — with gitsocial config site set <key> <value> --remote <name>, stored as machine-local git config (remote.<name>.gitsocial-site-{url,publish,pages}) next to the remote it describes. This is how one repo publishes to two buckets each stamping its own canonical URL, or a backup bucket carries data but no site (--remote backup publish false). Only these deployment keys are overridable; identity keys (title, description, accent, favicon) stay shared in the config ref. gitsocial config site get/list --remote <name> shows the effective merged view. See STATIC-SITE.md → Per-remote overrides.

No extra git configuration is needed when going through gitsocial; the helper is injected automatically. For bare git clone s3://… without gitsocial in the loop, run this once:

git config --global alias.remote-s3 '!gitsocial __git-remote-s3'

URLs & normalization

s3://<endpoint-host>/<bucket>/<prefix> is the only stored URL shape — identity and access in one string (cache, lists, and published data all carry it verbatim). A known provider's virtual-host spelling (s3://bucket.nyc3.digitaloceanspaces.com/repo) folds into it; bucket-only authorities and query params are errors.

clone / remote add additionally accept, as a boundary convenience only (protocol.ResolveS3URL, normalized before anything is stored):

Recognition is gated on the known host set, so a self-hosted endpoint still needs the explicit s3:// scheme, and an ordinary website URL passes through as a plain git remote.

Bucket layout

KeysContents
objects/<xx>/<38-hex>loose git objects, content-addressed
objects/pack/pack-<hash>.{pack,idx}packfiles and their indexes, written by pushes above the pack threshold and by sealing
refs/…, HEADone key per ref (or a .gen/<counter> chain in generation mode), plus the HEAD symref
info/refs, objects/info/packsthe dumb-HTTP transport surface: the ref listing in git update-server-info format and the pack list, so stock git clones the bucket over plain HTTPS
.gitsocial/gitsocial state: the ref-mode marker, the pack-sealing state, the thin-fork upstream marker, and the site's machine artifacts under .gitsocial/site/ (refs manifest, item/body/code indexes, pack map, configs, pages manifest, push-state marker)
index.html, gs-*.js (incl. gs-upgrade.js), pages-core.css, pages-full.css, icons.js, prism.js, grammars/the embedded site shell / app assets (when the site is enabled)
i/, issues/, prs/, posts/, releases/, memos/, sitemap.xml, sitemap-*.xml, robots.txtthe HTML page layer (when pages are enabled)
artifacts/<version>/…, artifacts/latest.txtrelease artifact objects and the newest non-prerelease version marker, owned by gitsocial release artifacts push (GITRELEASE.md §3.2 external storage; deliberately not under the page layer's releases/)

index.html is dual-owned: the generated front page when the HTML page layer is effective, the embedded SPA shell otherwise (the entry flip).

The shell filenames, the page-layer names, the artifacts/ prefix, and the dumb-HTTP transport keys (info/, objects/info/, and objects/pack/) form a reserved root namespace: no repo content may claim them. Site maintenance never deletes unrecognized root keys (its sweeps are name-listed or scoped to the reserved page namespaces), so artifact objects under artifacts/, the info/refs + objects/info/packs + objects/pack/ transport keys, and deployment-specific foreign objects (e.g. an installer script at the root) survive every push, pages regen, and pages-disable cleanup.

Dumb-HTTP clone

The bucket layout is already git's dumb-HTTP protocol: loose content-addressed objects at objects/<xx>/<38-hex>, plus HEAD and refs/heads/* / refs/tags/* as plain keys, all publicly served over HTTPS. Two derived keys complete it — stock git's dumb walker cannot list directories, so it needs them served:

So a bucket-served repo (behind its public domain) clones with stock git and no helper:

git clone https://<bucket-domain>/
git clone https://<bucket-domain>/ -c http.maxRequests=16    # all-loose bucket: parallelism instead of packs

On a packed bucket the walker reads objects/info/packs, fetches each .idx by convention (same basename) and then the .pack: a couple of downloads instead of one request per object, which is what makes the stock defaults fast. http.maxRequests still matters for a bucket whose history is all loose (small repos, or one not sealed yet) — but keep it modest: higher values can trip endpoint rate limits and fail the clone mid-walk (measured 2026-08-14 against a Cloudflare pub-*.r2.dev public dev endpoint: 16 works, 32 and 64 die with HTTP 429 "Cannot obtain needed tree … fetch failed"; custom domains may differ).

Both keys are mutable, so they are stamped no-cache (text/plain) and refreshed on every ref-moving push — a data-only push (site disabled) keeps them current too; gitsocial push --site-only refreshes them alongside the site read surface. A bucket first pushed by an older binary picks them up on its next ref-moving push.

Packfiles

A push uploads its object delta one of two ways, and never both for the same object: git's dumb walker prefers a loose key and only falls back to objects/info/packs on a 404, so an object duplicated in both is dead weight (no clone speedup, extra storage).

Delta sizeUpload
below GITSOCIAL_S3_PACK_THRESHOLD (default 1,000 objects)one loose key per object, as before
at or above ittwo packfiles

Two packs, split by type. Commits and tags pack with --depth=0 so no entry is a delta: the browser resolves any commit body (every commit is a user-visible item) from one byte range with no chain to walk, at a measured 0.5% size cost. Trees and blobs pack at git's default depth, where delta compression is the point. Packs are generational — a later push or sealing round writes a new pack and never repacks what is already packed — so recorded offsets stay valid forever.

State refs pack too. Objects reachable from refs/gitmsg/* (configs, list elements, forks, declines) pack and seal like everything else: every reader has a pack fallback (site maintenance resolves a packed config commit through the pack map with one Range GET; the browser probes both shapes). A bucket written by an older binary may still carry them loose until its next seal, which the loose fallback covers.

The pack map. .gitsocial/packmap/<xx>.json (one shard per two-hex sha prefix, brotli, mutable) maps each packed commit or tag to [packIndex, offset, size], so the browser reads an item body with a single Range GET and never touches a .idx. Trees and blobs have no map entry: the browser range-reads the .idx — its 256-entry fanout bounds the sha to one 1/256th slice of the sorted sha table, so a lookup reads a few hundred bytes rather than the whole index — then range-reads the pack and resolves OFS_DELTA / REF_DELTA chains itself, once per session and off the item hot path. The map deliberately stops at commits and tags: a shard holds 1/256th of everything it covers, so extending it to trees and blobs would grow each shard, and the whole-shard rewrite every push performs, in proportion to the object count the index searches in log time. See STATIC-SITE.md.

Fetching. gitsocial fetch and git fetch through the helper pull each listed pack once per session into GIT_DIR/objects/pack and run git index-pack locally, rather than resolving deltas in Go.

Sealing existing buckets. A bucket written before packing (or by pushes that all stayed below the threshold) carries its history loose. Post-push maintenance seals it when a seal is due — every 25 ref-moving pushes, or as soon as ~64 objects have gone loose since the last seal (a stock clone pays one request per loose object, so clone latency tracks the loose count, not the push count): it lists objects/, packs whatever is still loose, publishes the packs and their pack map, rewrites objects/info/packs, and records the round in .gitsocial/pack-state.json. An attempt whose sealable set is below the ~64-object minimum declines: it publishes nothing, advances the seal counter (the attempt paid the objects/ listing the counter rate-limits), and keeps the measured loose count so the next interval retries it.

A round's loose copies are deleted only once all three of these hold, and the round is deleted as one unit (a pack of the round missing from the bucket fails the whole round rather than collecting part of it):

GateWhy
3 further ref-moving pushesthe half of the grace nothing can skew, since the counter lives on the bucket
1 hour of wall clock since the seal returneda burst of pushes from several clones can otherwise burn a 3-push grace in seconds. Stamped when the pass returns, not when it starts, so a long first seal does not hand back a window it already spent. Advisory: the stamp comes from the sealing clone's clock and every other pusher compares it against its own, so a machine an hour fast treats a fresh round as expired on sight
the round's packs appear in a fresh read of objects/info/packsthat listing is the only way either reader (git's dumb walker, the browser) discovers a pack, and concurrent pushers rewrite it from whole listing snapshots, so one can drop the line another just published. Durable is not discoverable: deleting against an unadvertised pack is a 404 for every reader

This is the one exception to "content-addressed keys are never deleted" in the cache policy below: a loose object key can disappear once its content is available in a listed pack and the grace has passed. During the grace window the object exists in both places, which is deliberate and safe (the walker prefers loose).

Two more properties of the pass:

Pack upload is all-or-nothing: an interrupted push re-packs rather than resuming, and a pack that would exceed 4 GiB in a single PUT falls back to loose objects. Multipart upload is not implemented.

Ref update modes

Two modes, pinned per bucket by the .gitsocial/ref-mode marker (first writer decides from the endpoint host's provider capability; unknown hosts are probed):

ModeMechanismProviders
etagplain keys with If-Match CASaws, r2
generationcreate-only chains <ref>/.gen/<counter> via If-None-Match: *, highest generation wins, GC keeps the newest twodo

Readers resolve both shapes from one listing without the marker. Fast-forward is enforced client-side; buckets without conditional-write support are rejected before any ref is written.

git push --force-with-lease is supported in both modes: a matching lease authorizes a non-fast-forward update, while a lease invalidated by a concurrent pusher rejects with git's usual stale info.

Thin fork buckets

A fork published to your own bucket normally carries all of upstream's history. A thin bucket carries only the fork's own objects and records where a reader fetches the rest. Thinness is a property of the push relationship, recorded in local git config next to the remote it describes (exactly like the per-remote site overrides), never inferred:

KeyMeaning
remote.<name>.gitsocial-thintrue when pushes to this remote exclude upstream's objects
remote.<name>.gitsocial-upstreamthe upstream URL this relationship is thin against

A thin flag whose upstream is unset, or names a transport outside the allowlist below, degrades to a normal full push with a stderr note: nothing you push is ever secretly incomplete.

A thin bucket is helper-only. It is not a distribution surface:

SurfaceFull bucketThin bucket
gitsocial clone s3://… / git clone s3://… via the aliasworksworks (fetches upstream, then overlays)
git clone https://<bucket-domain>/ (stock git, no helper)worksfails at ref discovery, immediately and clearlyinfo/refs is deleted and never written
objects/info/packs, HEADwrittenwritten (the helper's own read path needs the pack list, and without a ref advertisement it grants no cloneability)
Static siteoptional, gated on site.publishrefused (PushSite and gitsocial push --site-only), any existing site left in place but unmaintained, with a stderr hint
Pack sealingrunsruns (it packs the fork's own loose objects, which is all a thin bucket has)
gitsocial fetch of registered fork metadataworksworks, with no upstream needed at all

gitmsg is never thinned. Objects reachable from refs/heads/gitmsg/* and refs/gitmsg/* are always pushed complete: they are empty trees and small commits (excluding them saves nothing measurable), and keeping them complete leaves the most-used fork read path — fork discovery, issues, PR metadata, comments, declines — free of any upstream dependency. The push splits its sources by ref class and widens the negative end for one of them:

code:   git rev-list --objects <code refs>   --not <bucket tips> <verified frontier>
gitmsg: git rev-list --objects <gitmsg refs> --not <bucket tips>

The frontier is verified by fetching, never by trusting local tracking refs (a tip upstream force-pushed past is fetchable by SHA only if the server sets uploadpack.allowAnySHA1InWant, which is policy, not protocol). Three tiers:

  1. git fetch --prune --no-tags <upstream> '+refs/heads/*:refs/remotes/upstream/*' succeeds ⇒ the frontier is the post-fetch tracking tips: advertised-now and locally present by construction. --prune drops branches upstream deleted.
  2. the fetch fails (offline, private, gone) ⇒ fall back to the pins the bucket already records, keeping those whose sha is present locally.
  3. neither ⇒ no exclusion at all: a full push, with a one-line stderr note.

Objects a dropped tip covered simply upload on the next push, so the fork fattens exactly as much as upstream drift requires.

The bucket key. One mutable key in the reserved .gitsocial/ namespace (no-cache under the cache policy), rewritten after every successful thin push:

.gitsocial/upstream
{
  "version": 1,
  "url": "https://github.com/alice/repo",
  "pins": [
    {"ref": "refs/heads/main", "sha": "…"},
    {"ref": "refs/heads/release-2.x", "sha": "…"}
  ]
}

Reading a thin bucket. The helper's fetch walk already stops at objects the local odb holds, so the overlay is a pre-fetch, not a resolver: on the first missing object the helper reads .gitsocial/upstream, validates the URL's transport (only https://, http://, s3:// — the key is bucket content, so a hostile bucket must not pick what git fetch executes; ssh is excluded because a thin bucket must record the URL its readers can fetch anonymously), fetches upstream into refs/remotes/upstream/*, retries any uncovered pin by sha, registers remote.upstream.url, and retries the object once. Upstream lands as a real remote so git gc cannot prune the borrowed objects. A commit neither the bucket nor upstream serves fails naming the commit and the upstream URL, not as a bare missing object.

Escape hatch. gitsocial push --full uploads every object the bucket lacks, restores info/refs, deletes .gitsocial/upstream, and clears remote.<name>.gitsocial-thin; the bucket is then an ordinary self-contained remote. gitsocial status shows Thin fork of <url>, pinned at <n> tips when the resolved push remote is thin.

Cache policy

Every upload is stamped with Cache-Control at the client (core/objstore/cache_control.go) so browser reloads and repeat visits are nearly free without ever serving stale ref state:

Key classCache-ControlEffect
Keys that never change once written: loose objects (objects/<xx>/<38-hex>), packfiles and their indexes (objects/pack/pack-<hash>.{pack,idx}), sealed site shards (.gitsocial/site/{items,bodies}/<ext>/shard-<hash>.json), sealed HTML list pages (<type>/<n>.html), sealed sitemap parts (sitemap-<n>.xml), release artifact objects (artifacts/<version>/<file>)public, max-age=31536000, immutableserved from disk, no network
Every mutable key (ref keys, HEAD, ref-mode, pack-state.json, the dumb-HTTP info/refs + objects/info/packs, the pack map shards, refs.json, the site artifact heads/manifests/cursor, the site shell, item pages, list heads, the generated index.html front page, sitemap.xml, robots.txt, artifacts/latest.txt)no-cachestore but always revalidate: a conditional If-None-Match GET yields 304 when unchanged, so the reader's ref-tip freshness comparison never reads a stale tip

The site reader uses the browser's default cache mode to honor these. The local dev servers (locals3, sitetest/serve.js) derive the same headers from the key pattern and answer conditional GETs, so local testing mirrors a real bucket.

The browser static site (the second surface above) reads this bucket layout directly and is documented separately in STATIC-SITE.md (site shell, sharded index/search artifacts, site config, and site testing).

Git helper discovery

Git spawns remote helpers as the subcommand git remote-s3, which resolves git aliases, so alias.remote-s3 = !gitsocial __git-remote-s3 is injected into gitsocial's own git invocations via environment config, and written to the local config of s3-origin workspaces and cloned repos so plain git works there. For bare git clone s3://… anywhere:

git config --global alias.remote-s3 '!gitsocial __git-remote-s3'

Local development

library/core/objstore/locals3 is a disk-backed local S3 server (stdlib-only) for development and tests; also used by the site-test fixture builder.

go build -o bin/locals3 ./library/core/objstore/locals3
bin/locals3 -root <dir>                        # 127.0.0.1:9000; each subdir of -root is a bucket

A loopback endpoint host (s3://localhost:9000/<bucket>/<prefix> or s3://127.0.0.1:9000/…) needs only dummy keys: loopback defaults to http + path-style, and any IP-literal host defaults to path-style. Non-loopback dev servers still take GITSOCIAL_S3_ENDPOINT=http://… + GITSOCIAL_S3_PATH_STYLE=1.

The same port serves the pushed static site browsably (extension-derived Content-Type, trailing-slash directory index, real-bucket cache headers): open http://127.0.0.1:9000/<bucket>/<prefix>/ after a push with the site guards on — no separate web server.

Manual testing (real providers)

Verified on AWS S3, Cloudflare R2, and DigitalOcean Spaces. Export credentials in the environment only (never in URLs or committed files); run one provider at a time.

Automated round trip — the acceptance bar per provider (capability probe → first push creating CAS → incremental push → clone back → gitmsg-ref SHA verification → fsck --strict, under a unique prefix with automatic cleanup):

GITSOCIAL_S3_TEST_HOST=<host> GITSOCIAL_S3_TEST_BUCKET=<bucket> \
  go test ./cli/gitsocial -run TestS3Helper_realBucket -v
# host: aws s3.<region>.amazonaws.com · r2 <account-id>.r2.cloudflarestorage.com · do <region>.digitaloceanspaces.com

Pass = provider round trip OK.

Zero-setup UX flow (fresh shell, only creds exported, no aliases or global config). Stored URLs are always the s3://<host>/<bucket>/<prefix> form; remote add/clone also accept a pasted AWS console URL and normalize it:

REMOTE=s3://<host>/<bucket>/manual-ux
git init uxdemo && cd uxdemo && git commit --allow-empty -m init
gitsocial remote add "$REMOTE"          # records the local `remote-s3` alias eagerly
gitsocial config site set publish true  # site guard (default off): enable the website
gitsocial social post "hello from a bucket"
gitsocial push                          # expect Pushed / Commits ≥ 1 / Site: published
git push origin main                    # plain git works via the recorded alias
cd .. && gitsocial clone "$REMOTE" uxclone && cd uxclone
git fsck --strict && gitsocial fetch && gitsocial social timeline   # expect the post
gitsocial push --site-only "$REMOTE"    # explicit site refresh (the push above already published it)

Then add the bucket as a timeline source from another workspace (social list create + social list add "$REMOTE" + fetch) and confirm the post appears.

Bucket shape (eyeball once): .gitsocial/ref-mode is etag (aws/r2, one plain key per ref) or generation (do, refs are .gen/<counter> chains; after ≥3 pushes to a ref at most the newest two generations remain).

Failure modes worth a check: a wrong-region host gives a clear signed-request error (not a hang); a bare s3://bucket/repo URL or any query param is rejected naming the required form; pushing generation-mode content to an etag-mode prefix errors loudly rather than spinning.

GITSOCIAL_S3_DEBUG=1 on any command dumps every request/response (with conditional headers and the provider's XML error body) to stderr. The automated test self-cleans; manual prefixes need manual deletion (console or a signed DELETE loop over the listed keys).

Credentials

Credentials resolve per endpoint host when a client is built, so a multi-remote push (gitsocial push r2 aws) signs each provider with its own keys in one invocation. Precedence:

  1. GITSOCIAL_S3_ACCESS_KEY / GITSOCIAL_S3_SECRET_KEY — explicit env override (both must be set)
  2. the ~/.config/gitsocial/credentials.json entry for the remote's endpoint host (honors XDG_CONFIG_HOME)
  3. AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — S3-ecosystem fallback (an AWS pair exported for other tooling never shadows a file entry)

The file is an object keyed by endpoint host — the canonical s3:// URL's host (<account>.r2.cloudflarestorage.com, s3.<region>.amazonaws.com, nyc3.digitaloceanspaces.com, 127.0.0.1:9000) — with {"accessKey": "...", "secretKey": "..."} values; provider/account granularity, no per-bucket keying. Manage it with the CLI (the key pair is read as two stdin lines, so it works both interactively and piped; the file is written with 0600, and looser permissions get a one-line stderr warning):

gitsocial config credentials set r2               # remote name → its endpoint host
gitsocial config credentials set 127.0.0.1:9000   # or a bare endpoint host
gitsocial config credentials list                 # hosts with masked access keys
gitsocial config credentials remove <host>

A missing file is silently fine (env vars still work); a malformed file warns once on stderr and falls through to the env fallback.

Environment variables

VariableMeaning
GITSOCIAL_S3_ACCESS_KEY / GITSOCIAL_S3_SECRET_KEYCredentials (take precedence over the credentials file)
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEYCredential fallback (S3-ecosystem convention; the credentials file beats it)
GITSOCIAL_S3_ENDPOINT / GITSOCIAL_S3_PATH_STYLEDev/self-hosted overrides: endpoint scheme (http) and path-style addressing
GITSOCIAL_S3_REGIONSigV4 region for endpoint hosts no preset recognizes
GITSOCIAL_S3_CONCURRENCYConcurrent object uploads per push (default 16, overrides the s3.concurrency setting). The reliable channel for the git-spawned helper subprocess
GITSOCIAL_S3_DEBUG=1Dump every request/response to stderr
GITSOCIAL_S3_PACK_THRESHOLDObject count at or above which a push uploads packfiles instead of loose objects (default 1,000). The sealing pass honors the same override for its own minimum (default ~64, the drift trigger). The site-test fixture lowers it to build a pack-only bucket
GITSOCIAL_SITE_SHARD_COUNT / GITSOCIAL_SITE_WALK_BUDGETTest-fixture overrides for the site artifacts: sealed-shard size (default 4000) and per-push walk budget (default 50000)
GITSOCIAL_SITE_PAGES_BUDGET / GITSOCIAL_SITE_SITEMAP_PARTTest-fixture overrides for the HTML page layer: per-push page budget (default 5000) and sitemap part size (default 40000)