S3 Remote Backend

An S3-compatible bucket is a git remote through the s3:// remote helper in the gitsocial binary (GITMSG.md §1.3), and it can serve the static site of the repository.

Clone · Push · Thin fork buckets · Local development · Testing · Reference

Clone

git clone https://<bucket-domain>/                                  # stock git, public bucket
gitsocial clone s3://<endpoint>/<bucket>/<prefix>                   # through the helper; private buckets work
git config --global alias.remote-s3 '!gitsocial __git-remote-s3'   # once, for bare `git clone s3://...`

• The bucket is a dumb-HTTP git repository, and stock git can clone it.
• gitsocial clone signs its reads with your credentials, so it can clone a private bucket. With no credentials it reads unsigned and can still clone and fetch a public bucket (see anonymous reads).
• A clone, and a workspace whose origin is a bucket, get the remote-s3 alias in their local git config. The global alias is for git clone s3:// in other directories.

Push

gitsocial config credentials set <endpoint-host>                        # paste the access key and the secret key
gitsocial remote add s3://<endpoint>/<bucket>/<prefix> --default --site  # default push target, site on
gitsocial push

• URLs are stored as s3://<endpoint-host>/<bucket>/<prefix>. remote add and clone also take a provider's https:// endpoint or virtual-host URL and a pasted AWS console URL. A self-hosted endpoint needs the s3:// form.
• --default appends the remote to gitsocial.pushRemote; --site sets site.publish (see STATIC-SITE.md).
• An s3:// URL added with plain git remote add works too.
• Credentials are stored per endpoint host. credentials list shows them with masked keys, and credentials remove <host> deletes one.

gitsocial push [remote...] sends the gitmsg/* branches, the refs/gitmsg/* state refs, all tags, the default branch when it is ahead, and the heads of open pull requests. A push to an s3 remote with site.publish then rebuilds the site.

• Remotes resolve in this order. The TUI's p and core.push over RPC resolve the same list.
  1. the arguments
  2. gitsocial.pushRemote, set with gitsocial remote default <name...>
  3. origin, if it is an s3 remote
  4. the first s3 remote in alphabetical order
  5. origin
• --all-remotes skips the resolution and pushes to every configured remote in alphabetical order.
• Several remotes push in sequence. A failed remote is reported and skipped, and the command exits non-zero.
• Diverged gitmsg/* branches merge automatically. Diverged code branches fail with a hint.
• A plain git push to an s3 remote also rebuilds the site when the pushed config ref sets site.publish.

Output line	Meaning
Pushed to <remote> (<url>)	the refs are on the remote
Site: published	the site rebuild is complete
Site: published (incomplete: ...)	the first build of the site (the bootstrap) continues over more pushes; push again to finish
Site: published (fork pull requests: <n>, <size> MB)	the fork pull request area is on, with its count and size
Site: skipped (<reason>)	no site rebuild for this push
Site: failed: <err>	the rebuild failed after the refs were sent; a warning, not a push failure

Flag	Effect
--no-code	skip code branches
--all-branches	send every local branch; a non-fast-forward branch is reported and not merged; reconcile with git push --force-with-lease
--all-remotes	push to every configured remote in alphabetical order; takes no remote arguments
--no-site	skip the site rebuild for this push
--site-only	rebuild the site, send no refs
--full	send every object and detach a thin fork bucket
--dry-run	print the plan, send nothing, rebuild nothing, stay offline

url, publish and pages can be different for each remote (see per-remote overrides).

Thin fork buckets

A fork bucket usually holds all the history of its upstream. A thin bucket holds only the objects of the fork and records the URL of the upstream that has the other objects. gitsocial fork create with a bucket destination sets up a thin bucket; to set one up manually, set two keys on the remote:

git config remote.<name>.gitsocial-thin true
git config remote.<name>.gitsocial-upstream https://github.com/org/repo

• A thin push fetches the upstream first and does not upload the objects that the upstream has. When the upstream is not reachable, the push prints a note and skips only the objects that the bucket already has.
• The bucket gets gitmsg/* and refs/gitmsg/* complete, so issues, comments and pull requests are readable without the upstream.
• The upstream must be https://, http:// or s3://; ssh:// is not accepted. With another scheme or no upstream, the push is a full push and prints a note. The helper checks the scheme before it fetches.
• When a clone needs an object that the bucket does not have, it fetches from the upstream recorded in .gitsocial/upstream. The fetch runs one time per session into refs/remotes/upstream/* and registers upstream as a remote, so the borrowed objects stay reachable and a later git gc does not remove them.
• --full uploads the missing objects as loose objects, not as a packfile; the next seal packs them into packfiles.
• gitsocial status shows Thin fork of <url>, pinned at <n> tips.
• gitsocial push --full detaches the bucket: it uploads every object, restores info/refs, deletes .gitsocial/upstream and clears the flag.

Surface	Full bucket	Thin bucket
gitsocial clone s3://, git clone s3:// through the alias	works	works, and fetches the missing objects from the upstream
git clone https://<bucket-domain>/	works	fails at ref discovery; info/refs is not written
Static site	rebuilt on site.publish	not rebuilt; an existing site stays, and is not updated
gitsocial fetch of fork metadata	works	works without upstream

Local development

go build -o bin/locals3 ./library/core/objstore/locals3
bin/locals3 -root <dir>          # 127.0.0.1:9000; each subdirectory of -root is a bucket

A loopback host (s3://127.0.0.1:9000/<bucket>/<prefix>) needs only dummy keys and uses http and path-style by default; every other IP-literal host also uses path-style by default. For another development server, set GITSOCIAL_S3_ENDPOINT=http://... and GITSOCIAL_S3_PATH_STYLE=1. The same port serves a pushed site at http://127.0.0.1:9000/<bucket>/<prefix>/.

locals3 follows S3 on byte ranges, listing pages of 1,000 keys, If-Match and keys that prefix other keys.

gitsocial serve runs the same bucket handler in the process for the workspace and serves its site; see CLI.md.

Testing against a provider

Verified on AWS S3, Cloudflare R2 and DigitalOcean Spaces. Export credentials in the environment only, one provider at a time.

GITSOCIAL_TEST_FULL=1 GITSOCIAL_S3_TEST_HOST=<host> GITSOCIAL_S3_TEST_BUCKET=<bucket> \
  go test ./cli/gitsocial -run TestS3Helper_realBucket -v

Hosts: s3.<region>.amazonaws.com, <account-id>.r2.cloudflarestorage.com, <region>.digitaloceanspaces.com. A pass prints provider round trip OK, and the test deletes the keys that it writes.

The manual flow from a new shell, with credentials exported and nothing else configured:

git init uxdemo && cd uxdemo && git commit --allow-empty -m init
gitsocial remote add "$REMOTE" && gitsocial config site set publish true
gitsocial social post "hello from a bucket" && gitsocial push       # Pushed, Commits 1, Site: published
git push origin main                                                 # plain git through the recorded alias
cd .. && gitsocial clone "$REMOTE" uxclone && cd uxclone
git fsck --strict && gitsocial fetch && gitsocial social timeline    # the post is there

GITSOCIAL_S3_DEBUG=1 dumps every request and response to stderr. The manual flow does not delete its prefixes, so delete them yourself.

Reference

Bucket layout

Keys	Contents
objects/<xx>/<38-hex>	loose git objects
objects/pack/pack-<hash>.{pack,idx}	packfiles and their indexes
refs/..., HEAD	one key per ref, or a .gen/<counter> chain in generation mode, plus the HEAD symref
info/refs, objects/info/packs	the dumb-HTTP ref listing and pack list, rewritten on every ref-moving push
objects/info/http-alternates, objects/info/alternates	present and empty; a 404 on them can livelock the dumb walker of git
.gitsocial/refs.json	the ref manifest: refname to sha for every ref in the bucket; written before and after each ref-moving push and by gitsocial push --site-only; the ref source for clients that cannot list (anonymous or not) and for the static site
.gitsocial/ref-mode, .gitsocial/pack-state.json, .gitsocial/packmap/<xx>.json, .gitsocial/upstream	the ref mode, the sealing state, the pack map, the thin-fork marker
the shell files at the root, the page directories, .gitsocial/site/	the static site; the page and artifact keys are in STATIC-SITE.md
.gitsocial/forks/	the fork pull request area: the manifest, one pack per open head and the lock marker
artifacts/<version>/..., artifacts/latest.txt	release artifacts and the newest version marker, written by gitsocial release artifacts push

These names are reserved, and repository content does not use them. The site rebuild deletes only the keys that it wrote, so release artifacts and other keys at the root stay after each push.

Ref update modes

The first push records the mode in .gitsocial/ref-mode from the provider of the endpoint host, or from a probe of the bucket when the host is unknown. The helper refuses a bucket without conditional writes before it writes a ref.

Mode	Mechanism	Providers
etag	one key per ref, updated with If-Match	AWS, R2
generation	create-only chain <ref>/.gen/<counter>; the highest generation is the value; the newest two are kept	Spaces

The helper enforces fast-forward. git push --force-with-lease works in both modes.

Packfiles

Objects in a push	Upload
under GITSOCIAL_S3_PACK_THRESHOLD (default 1,000)	one loose key per object
at or above it	two packfiles: commits and tags at --depth=0, trees and blobs at git's default depth

Maintenance attempts a seal on the first ref-moving push to a bucket, then every 25 pushes, and also when 64 or more objects were uploaded loose after the last attempt. A push that defers maintenance under GITSOCIAL_S3_DEFER_MAINTENANCE does not add its loose objects to that count, so one gitsocial push of several branches counts only the objects of its last git push. After a seal, its loose copies are deleted after 3 more pushes and 1 hour. A shallow or partial clone does not seal.

The client has no multipart upload, so when a pack of a push is over 4 GiB, the push uploads all its objects as loose objects and packs no part of the push. A seal instead skips a pack over 4 GiB (see Sealing).

The pack of commits and tags is written at --depth=0, so one ranged read of a commit gives a complete object. A push of 2,000 objects or more first lists objects/ and skips the objects that the bucket already has, so an interrupted first push continues where it stopped. The pack-objects command line sets pack.indexVersion=2 and pack.packSizeLimit=0, and the git config of the pusher does not change them.

The byte range of each object in a pack is published in the pack map. When a shard of the pack map cannot be written, the push logs the error and continues; a client with no pack map entry for an object reads the range from the .idx.

The pack index readers of the helper and of the site app read 20-byte object ids, so a repository with SHA-256 object ids is not supported: neither reads a pack of its bucket.

Sealing

A seal packs the loose objects of the bucket into packfiles, and the packs of one seal are a round. After a grace period, maintenance deletes the loose copies of the objects in the round, which is the only deletion of an immutable key. Each step is idempotent; a failure leaves the bucket as it was, and the next push tries again.

• A seal packs the objects of the bucket ref tips that the pushing clone can resolve; a ref that the clone does not have is skipped and does not cause a failure.
• .gitsocial/pack-state.json is written with compare-and-swap, so when two pushes finish at the same time, both rounds are kept. A state document at another schema version is an error, and the push does not overwrite it.
• Maintenance deletes the loose copies of a round only after a new read of objects/info/packs lists the packs of the round; stock git and the helper find packs through that list.
• A seal skips a pack over 4 GiB and publishes the other packs; the objects of the skipped pack keep their loose keys, and the next seal tries them again.
• The grace period is a push count and a wall-clock hour, and both must pass before a loose copy is deleted.
• The loose copies of a round are deleted as one unit: when a pack of the round is not in the bucket, the deletion fails and no loose copy is deleted.
• Each seal writes new packs and does not change the packs of earlier rounds, so the number of packfiles increases with each seal; nothing merges them.
• A failed seal records its error in the bucket, and each later push prints that error until a seal succeeds.

Push maintenance

Maintenance is the work that the helper does on the bucket after each ref-moving push. It runs after the push report, and its errors do not fail or delay the git push.

• The helper sends the per-ref report and its final blank line to git before maintenance starts.
• One gitsocial push is a sequence of git pushes; all but the last set GITSOCIAL_S3_DEFER_MAINTENANCE, so maintenance runs one time, on the final refs. A plain git push sets nothing, so maintenance runs each time.
• Maintenance sets the bucket HEAD to the HEAD symref of the repository, not to an assumed main, and keeps an existing HEAD unless it points at a gitmsg/* branch. When HEAD names a ref that is not in the bucket, maintenance points it at a branch that is in the bucket.
• info/refs and objects/info/packs are rewritten on each ref-moving push, also when the site is off.
• The site part of maintenance, and the marker that lets it skip, are in STATIC-SITE.md.
• Each mutable JSON document that two pushes can write is written with compare-and-swap. The helper reads the key with its ETag, merges, and writes only if the ETag did not change; on a conflict, it merges again. A provider without conditional updates gets an unconditional write, and the last write has priority. A document that is present but does not parse is a read error, not an absent document.

Fetch

gitsocial clone and gitsocial fetch walk the object graph from the wanted tips and write objects directly into GIT_DIR/objects. The bucket stores loose objects in the git format, so a download is a byte-for-byte copy.

• When the bucket has packfiles, the client downloads and indexes them with git index-pack one time per session, and the walk then reads those objects from the local object database. A pack is downloaded under a temporary name and gets its final name only after index-pack completes.
• When the local object database already has an object, the walk assumes that its graph is complete and does not descend into it, the same assumption that the haves negotiation of git makes. An object from a downloaded packfile is the exception: the walk descends through it.
• The empty tree is not checked for presence, and the walk downloads it from the bucket; cat-file finds the empty tree also when the local object database has no copy. Each gitmsg/* commit has the empty tree.
• On a bucket that can list, the client can get the value of a ref without a GET when the ref manifest gives a sha and the ETag of the ref key in the listing is the MD5 of <sha>\n. For a multipart ETag, a different ETag, or a generation chain, the client reads the ref key.

Cache policy

Every upload carries Cache-Control. The local servers derive the same headers from the key pattern.

Keys	Cache-Control
loose objects, packfiles and indexes, release artifacts, the versioned shell directories (.gitsocial/site/shell/<rev>/), and the site's full shards, full pages and sitemap parts	public, max-age=31536000, immutable
every mutable key: refs, HEAD, the .gitsocial/ state, info/refs, objects/info/packs, artifacts/latest.txt, and every other site key	no-cache

The helper does not rewrite immutable keys and deletes only a loose object after a seal packed it and the grace period passed.

Each PUT sets Cache-Control from the key; the header is not signed and does not change the SigV4 signature. Three places apply the table above: cacheControlForKey for each upload, siteCacheControl for the immutable site keys, and localbucket (the handler behind locals3), which calculates the header again on each read.

Request behavior

All reads and writes retry in one place, and no caller has its own retry loop.

Outcome	Behavior
429, any 5xx, the 400 AWS documents as a retryable stalled upload, a transport failure carrying no status	3 attempts, 0.5 s then 2 s apart
404, 403, a precondition failure, every other 4xx	a definite answer, returned immediately

• Writes retry the same as reads; a PUT of the same key and the same bytes is idempotent.
• A listing that reports truncation with no continuation token is an error.
• A CDN that compresses a response marks its ETag as weak, and the client removes the W/ prefix before it uses the value.
• The HTTP transport keeps 64 idle connections per host for the concurrent uploads.
• Object keys are signed with the AWS URI encoding, so a key with +, @, = or , is signed as it is in the request line.

Thin bucket marker

.gitsocial/upstream is rewritten by a thin push that computed a frontier: the upstream branch tips from its fetch of the upstream, each tip a pin. A push that computed no frontier keeps the stored marker when it names the same upstream, and writes a marker with no pins when the upstream changed.

{"version": 1, "url": "https://github.com/alice/repo", "pins": [{"ref": "refs/heads/main", "sha": "..."}]}

Credentials

Resolved per endpoint host, in this order: GITSOCIAL_S3_ACCESS_KEY with GITSOCIAL_S3_SECRET_KEY, then ~/.config/gitsocial/credentials.json, then AWS_ACCESS_KEY_ID with AWS_SECRET_ACCESS_KEY. A source counts only when both keys are set, so a single variable is skipped and is not mixed with another source. When no source is complete, the client is anonymous.

The file path honors XDG_CONFIG_HOME. The file is an object keyed by endpoint host (<account>.r2.cloudflarestorage.com, s3.<region>.amazonaws.com, 127.0.0.1:9000) with {"accessKey": "...", "secretKey": "..."} values. It is written with mode 0600, and a less strict mode gives a warning. A missing file is not an error; a malformed file gives one warning, and the client continues to the AWS variables.

Anonymous reads

With no credentials for the endpoint host, the client is anonymous: it signs nothing and reads only. A push stops with the credentials error before it uploads anything.

A public bucket usually grants s3:GetObject but not s3:ListBucket, so ref discovery cannot list refs/ and takes the ref names from the first document that the bucket has: .gitsocial/refs.json, then info/refs. Also, an absent key answers 403, not 404, so the client reads a 403 on a read as "not found", and thin-fork borrowing and the pack probes still work. A signed client without s3:ListBucket does the same. On a signed GET, a 403 is "not found" only when the provider reports AccessDenied, so a rejected credential keeps its own error; each signed HEAD 403 is "not found", because a HEAD response has no body.

• In etag mode, the client reads the key of each ref for its value, so an old manifest can omit a ref or name a deleted ref but cannot give an incorrect sha. In generation mode, only a listing finds the tip of a chain, so the manifest gives the value, and a ref deleted after the last push stays visible until the next push.
• Each ref-moving push rewrites the manifest from its own refs. Before a push, the helper lists the refs and creates or corrects the manifest, so a push that moves no ref also leaves a current manifest. This write occurs before git decides if there is anything to push.
• In etag mode, the manifest write is conditional on the manifest from before the push: when a concurrent push changed it, the helper lists the refs again and does not overwrite it. A generation-mode bucket cannot write conditionally, so two pushes at the same time can leave the manifest one push old until the next push. A failed manifest write has the same result when the next transfer moves no ref.
• A document that is present and empty means zero refs. Discovery stops there and does not read an older source.
• A bucket that was last pushed before .gitsocial/refs.json existed is read through info/refs. A thin fork bucket has neither file and needs one push from a current gitsocial first.
• When no ref document is readable, the client reads .gitsocial/ref-mode and HEAD. If they are readable, the bucket needs one push from a current gitsocial; if not, the bucket is private or empty, and the error names the credentials to set.
• Anonymous reads need no s3:ListBucket, and adding it also works.

Environment variables

Variable	Meaning
GITSOCIAL_S3_ACCESS_KEY, GITSOCIAL_S3_SECRET_KEY	credentials; both required; priority over the file
neither pair set, and no file entry	anonymous reads: unsigned, read-only
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY	credentials, used after the file
GITSOCIAL_S3_ENDPOINT, GITSOCIAL_S3_PATH_STYLE	endpoint scheme and path-style addressing for self-hosted servers
GITSOCIAL_S3_REGION	SigV4 region for a host no preset recognizes
GITSOCIAL_S3_CONCURRENCY	concurrent uploads per push; default 16; overrides the s3.concurrency setting
GITSOCIAL_S3_DEBUG=1	dump every request and response to stderr
GIT_QUIET	set by git; silences the remote helper's progress on stderr
GITSOCIAL_CACHE_DIR	the cache directory of the remote helper; gitsocial sets it from --cache-dir for the git it runs, so a plain git push from another shell reads the default cache
GITSOCIAL_S3_PACK_THRESHOLD	objects per push at which a push uploads packfiles, default 1,000; also the minimum count of loose objects that a seal packs, default 64
GITSOCIAL_SITE_SHARD_COUNT, GITSOCIAL_SITE_WALK_BUDGET	test-fixture overrides: full shard size (default 4,000) and per-push walk budget (default 50,000)
GITSOCIAL_SITE_PAGES_BUDGET	limit on item pages written per push; unbounded when unset
GITSOCIAL_SITE_SITEMAP_PART	test-fixture override: sitemap part size, default 40,000