Single Go library with thin clients: CLI/TUI (direct) and JSON-RPC (stdio).
Development • Directory Structure • Package Reference • Cache Architecture • S3 Remote Backend • CLI Commands • TUI
Trunk-based: main is the integration branch and stays clean. Develop each feature on a feature/<name> branch; rebase on main and fast-forward merge so history stays linear.
git switch main && git pull --ff-only # refresh before branching and after any merge
git switch -c feature/<name> # start a feature
git fetch origin && git rebase main # keep current with main
git switch main && git merge --ff-only feature/<name> # integrate
git branch -d feature/<name> # clean up
gitmsg/* and gitsocial are protocol/data branches — not for feature work.Build, test & run:
go build -o bin/gitsocial ./cli/gitsocial # Build CLI
go build -o bin/ ./... # Compile-check everything (all mains land in bin/, never the repo root)
go test ./... # Run tests
bin/gitsocial social timeline # Run command
bin/gitsocial tui # Launch TUI
main's binary don't clobber each other (mains land in bin/, never the repo root).--cache-dir (e.g. --cache-dir /tmp/gs-<name>): the first binary to open the shared ~/.cache/gitsocial/cache.db upgrades it in place, after which older binaries (other branches, main) refuse it. Delete the cache to rebuild.The gate is scripts/check.sh, in two tiers. scripts/check.sh --quick runs the prose check, go vet, golangci-lint and every test except the guarded ones; the pre-push hook runs it on every push (about 90 s warm when every package reruns, less after a change in one package). scripts/check.sh runs the same with GITSOCIAL_TEST_FULL=1, so the guarded tests run too (about 4 min warm); run it before merging to main and at release. Stages run in order, the failing stage is named, and any failure exits non-zero. A missing golangci-lint fails unless --skip-lint is passed.
Guarded tests call fullTierOnly: the TUI matrices TestSmoke, TestSequence and TestGolden/LayoutProperties, the CLI --json walk TestCommandTreeJSONOutput, and the TestS3Helper_* child-process tests. -race and the browser site battery run at release (scripts/release.sh preflight). -short skips 66 real-git subtests and is a local smoke run, never a tier.
scripts/prose-check.sh is stage 0. It counts STYLE.md violations (em-dashes, comment blocks over three lines, long Short and flag help) and fails when a count rises above scripts/prose-baseline.txt; --update accepts the current counts, --list <rule> prints the offending lines. Commit subjects over 72 characters in the pushed range fail outright.
scripts/check.sh --quick # the push tier
scripts/check.sh # the full tier
scripts/check.sh --skip-lint # golangci-lint not installed (not a gate run)
scripts/check.sh -short ./... # extra args go to the test stage (smoke run, not a gate run)
git config core.hooksPath scripts/hooks # install the pre-push hook (once per clone)
GITSOCIAL_SKIP_GATE=1 git push # skip the gate once
Coverage (scripts/coverage.sh, writes .test-artifacts/coverage/): runs go test ./... -coverpkg=./... so code exercised by another package's integration tests gets credit, then reports the statement-weighted total (not a mean of per-package percentages), a ranked per-package table including packages with no test files of their own, and every function at 0.0%. The total is a floor: coverage cannot see the S3 helper tests (they run the helper as a child process, so helper_push.go / thin.go look untested) or the ~40 browser suites behind -tags sitetest (site_pages*.go).
go test ./... # All tests
go test ./library/core/cache # Specific package
go test -v ./... # Verbose output
go test -race ./... # Race detector
go test -cover ./... # Coverage summary
golangci-lint run --fix ./... # Lint & fix code
scripts/test.sh # Streams per-test progress; wraps `go test -json` (accepts the same args, e.g. `scripts/test.sh -race ./...`)
go test ./library/tui/test/... # Headless TUI suite (smoke/display/golden/nav/sequence; see TUI-TESTS.md)
go test -tags sitetest -timeout 30m ./library/core/objstore/ # Static-site browser suites (wraps scripts/site-test.sh; needs node; see STATIC-SITE.md)
library/extensions/* → library/core/* → stdlib only
↓ ↓
cli/gitsocial/ (no circular refs)
library/tui/
library/rpc/
library/import/ → library/extensions/* + library/core/protocol
specs/GITMSG.md, specs/GITSOCIAL.md, specs/GITPM.md, specs/GITRELEASE.md, specs/GITREVIEW.mddocumentation/STYLE.md (prose, help text, comments, commits)documentation/ files// commits.go - Git commit operations)fmt.Errorf("context: %w", err)cache.ExecLocked/QueryLocked for all DB operationscore/git)Core packages (cache, git, protocol) → return error (idiomatic Go)
Extension public API (social.*) → return Result[T] (user-facing codes)
Internal helpers → return error
Rules:
fmt.Errorf("operation: %w", err)Result[T] for user-friendly error codescache.QueryLocked / ExecLocked; example: library/extensions/social/.Result[T], built with result.Ok / result.Err; example: library/extensions/social/.*cobra.Command per file under cli/gitsocial/, registered in init().View interface (Update/Render); example: library/tui/tuicore/.gitsocial/ # module github.com/gitsocial-org/gitsocial
├── cli/gitsocial/ # CLI thin client; builds the binary
├── library/ # Go library — single source of truth
│ ├── core/ # Shared infrastructure
│ │ ├── git/ # Git operations
│ │ ├── protocol/ # GitMsg protocol parsing
│ │ ├── gitmsg/ # Protocol-level storage
│ │ ├── cache/ # SQLite operations
│ │ ├── storage/ # Bare repo management
│ │ ├── objstore/ # S3 remote backend (stdlib client + git remote helper)
│ │ ├── fetch/ # Fetch orchestration + processing
│ │ ├── identity/ # Identity declarations + verification
│ │ │ └── forge/ # Forge adapters (GitHub, …)
│ │ ├── notifications/ # Notification aggregation
│ │ ├── search/ # Cross-extension search
│ │ ├── settings/ # User settings + config paths
│ │ ├── log/ # Structured logging
│ │ ├── text/ # String helpers
│ │ └── result/ # Result[T] type
│ ├── extensions/
│ │ ├── social/ # Posts, lists, timeline
│ │ ├── pm/ # Issues, milestones, sprints
│ │ ├── release/ # Releases, versions, artifacts
│ │ ├── review/ # Pull requests, code reviews
│ │ └── memo/ # Tiered memos (knowledge as commits)
│ ├── proposals/ # Cross-repo proposals (gate; accept + decline engine)
│ ├── import/ # Platform import pipeline
│ │ ├── github/ # GitHub adapter (gh CLI)
│ │ └── gitlab/ # GitLab adapter
│ ├── clientfetch/ # Thin-client fetch orchestration (CLI/TUI)
│ ├── rpc/ # JSON-RPC server (stdio) — thin-client surface
│ └── tui/ # TUI views — thin client
├── documentation/ # Protocol + architecture docs
├── scripts/ # Build/release/test scripts (release.sh, install.sh, test.sh, site-test.sh)
└── specs/ # Protocol specifications
Outside the repo tree:
~/.config/gitsocial/ # User config; honors `XDG_CONFIG_HOME`
├── settings.json # Machine-specific settings
└── personal/ # Personal-tier bare repo (override: `GITSOCIAL_PERSONAL_REPO`)
~/.cache/gitsocial/ # Cache dir, `--cache-dir` overrides
├── cache.db # SQLite (commits + extension tables)
├── repositories/ # Bare git clones (cleaned up periodically)
├── forks/ # Fork bare clones
├── imports/ # Import mapping files (per repo URL slug)
└── memo/session/ # Per-session memo bare repos
| Package | Key Types | Key Exports |
|---|---|---|
core/gitGit operations | Commit, FileDiff, Hunk, DiffLine, DiffStats | GetCommits, CreateCommit, ReadRef, WriteRef, GetDiff, GetFileDiff, GetFileContent, GetDiffStats, MergeBranches, SquashMerge, RebaseMerge, ForceMerge, RebaseBranch, RangeDiff, PatchesEqual, GetBehindCount, GetMergeBase, GetUserName, GetGitConfig, CreateSignedCommitTree, VerifyCommitSignature, GetCommitSignerKey |
core/protocolMessage parsing | Header, Message, Origin, Trailer | ParseMessage, ParseHeader, CreateHeader, FormatMessage, ParseRef, CreateRef, FormatShortRef, QuoteContent, ApplyOrigin, ExtractTrailers, Trailer, IsClosingTrailer |
core/cacheSQLite operations | Repository, Commit, TrailerRef | Open, DB, ExecLocked, QueryLocked, InsertCommits, FilterUnfetchedCommitsByRepo, MarkCommitsStaleByRepo, ResetRepositoryData, RegisterMigration, ToNullString, ToNullInt64, GetTrailerRefsTo, TrailerRef |
core/gitmsgProtocol-level storage | — | ResolveRepoURL, Push, ReadExtConfig, WriteList, GetHistory, GetExtBranch, IsExtInitialized, GetForks, AddFork, AddForks, RemoveFork |
core/storageBare repo management | — | EnsureRepository, GetStorageDir, FetchRepository |
core/objstoreS3 remote backend (see S3 Remote Backend) | Client, Config, Capability, HelperEnv | NewClient, ParseS3URL, RunHelper, HelperEnvFromOS, ListRemoteRefs, PushSite, PushArtifactObjects, PutObjectToRemote |
core/fetchFetch orchestration | — | FetchAll, FetchRepository, FetchForks, CommitProcessor, PostFetchHook |
core/settingsUser settings | — | Get, Set, ListAll |
core/searchCross-extension search | — | Search, Params, Result, Item, Group, GroupedItem, FormatResult, IsValidGroupBy |
core/resultResult type | Result[T], Error | Ok, Err, ErrWithDetails |
core/notificationsNotification aggregation | Notification, Provider, Filter | RegisterProvider, GetAll, GetUnreadCount, MarkAsRead, MarkAsUnread, MarkAllAsRead, MarkAllAsUnread, MentionProcessor, ExtractMentions, TrailerProcessor |
core/identityIdentity verification | Identity, ResolvedIdentity, DNSIdentity, Binding, Source, VerifyCandidate | VerifyBinding, IsVerified, IsVerifiedCommit, LookupBinding, VerifyCandidates, NormalizeSignerKey, NormalizeEmail, ResolveIdentity |
core/identity/forgeForge adapters for identity verification | Forge, GPGKey, CommitVerification | Forge, Register, Lookup, LookupForRepo, ParseRepoURL, NewGitHub, GPGKey, CommitVerification |
extensions/socialSocial layer | Post, SocialItem | GetPosts, CreatePost, GetTimeline, Fetch |
extensions/pmProject management | Issue, Milestone, Sprint, PMNotification | GetIssues, CreateIssue, GetMilestones, GetSprints, MessageToPMItem, FetchRepository, Processors |
extensions/releaseRelease management | Release, ReleaseItem, ReleaseNotification | CreateRelease, EditRelease, GetReleases, GetSingleRelease, MessageToReleaseItem, FetchRepository, Processors |
extensions/reviewCode review | PullRequest, Feedback, ReviewSummary, StackEntry, ReviewNotification | CreatePR, GetPR, UpdatePR, MergePR, ClosePR, RetractPR, MarkReady, ConvertToDraft, UpdatePRTips, SyncPRBranch, GetPRVersions, ComparePRVersions, GetVersionAwareReviews, CreateFeedback, GetReviewSummary, MessageToReviewItem, FetchRepository, GetPullRequestsWithForks, GetStack, GetDependents, Processors |
extensions/memoTiered memos (knowledge as commits) | Memo, MemoItem, Tier, SessionInfo | CreateMemo, EditMemo, RetractMemo, PromoteMemo, ListMemos, GetSingleMemo, InitProject, InitPersonal, InitSession, ListSessions, GCSession, PushPersonal, FetchPersonal, PushSession, FetchSession, SyncAllTierReposToCache, AddInherit, RemoveInherit, ListInherits, IsInherited |
proposalsCross-repo proposals | Outcome | Accept, Decline |
importPlatform import pipeline | SourceAdapter, ImportPlan, Stats, MappingFile | Run, SourceAdapter, ReadMapping, WriteMapping, MappingKey, ResolveHost, MapLabels |
import/githubGitHub adapter | — | New, CheckGH, Adapter.FetchPM, Adapter.FetchReleases, Adapter.FetchReview, Adapter.FetchSocial |
| Term | Context | Meaning |
|---|---|---|
original | GITSOCIAL field | Post being commented/reposted/quoted |
canonical | Versioning | First version of a message (before edits) |
edits | GITMSG field | Reference to canonical version being edited |
Key principle: Storage (repositories/) can be deleted anytime. Fetch strategy is determined by cache.db metadata, not storage state.
Staleness: The cache is append-only, but commits that no longer exist in their source branch (e.g., after rebase or force-push) are marked with a stale_since timestamp via cache.MarkCommitsStale() (single-branch) or cache.MarkCommitsStaleByRepo() (all-branch). Stale commits are excluded from timeline and list queries but remain visible (dimmed) in thread and detail views to preserve discussion context.
SQLite tuning: WAL mode, 64MB cache (_cache_size=-65536), memory temp store, 16 max connections, 256MB mmap.
Core tables:
core_commits(repo_url, hash, branch, author_name, author_email, message, timestamp, edits, is_virtual, origin_author_name, origin_author_email) - PK: (repo_url, hash, branch)core_commits_version(edit_repo_url, edit_hash, edit_branch, canonical_repo_url, canonical_hash, canonical_branch, is_retracted) - PK: (edit_repo_url, edit_hash, edit_branch)core_repositories(url, branch, storage_path, is_followed, last_fetch) - PK: urlcore_lists(id, name, source, version, workdir) - PK: idcore_list_repositories(list_id, repo_url, branch) - PK: (list_id, repo_url)core_fetch_ranges(id, repo_url, range_start, range_end, status, fetched_at, commit_count, error_message)core_notification_reads(repo_url, hash, branch, read_at) - PK: (repo_url, hash, branch)core_mentions(repo_url, hash, branch, email) - PK: (repo_url, hash, branch, email)core_trailer_refs(repo_url, hash, branch, ref_repo_url, ref_hash, ref_branch, trailer_key, trailer_value) - PK: (repo_url, hash, branch, ref_repo_url, ref_hash, ref_branch, trailer_key)core_identity_dns(email, key, repo, resolved_at) - PK: email — caches DNS well-known lookups (24h TTL).core_verified_bindings(key_fingerprint, email, source, forge_host, forge_account, verified, resolved_at) - PK: (key_fingerprint, email, source, forge_host) — caches per-source attestations. See Identity Verification for the trust model and source list.core_edit_acceptances(edit_repo_url, edit_hash, edit_branch) - PK: (edit_repo_url, edit_hash, edit_branch) — derived index that a cross-repo proposal was accepted, populated from the mirror edit's accepts= header on every fetch path. Read only as a NOT EXISTS marker (clears the proposed-edit ✎) and for accept idempotency.core_edit_declines(edit_repo_url, edit_hash, edit_branch) - PK: (edit_repo_url, edit_hash, edit_branch) — the owner declined a cross-repo proposal. Durable and published at refs/gitmsg/core/declines/* so the proposer learns and the choice survives a re-clone; clears the proposed-edit marker (accept takes precedence).Social extension:
social_items(repo_url, hash, branch, type, original_*, reply_to_*) - PK: (repo_url, hash, branch)social_interactions(repo_url, hash, branch, comments, refs) - PK: (repo_url, hash, branch)social_followers(repo_url, workspace_url, detected_at, list_id, commit_hash) - PK: (repo_url, workspace_url)Release extension:
release_items(repo_url, hash, branch, tag, version, prerelease, artifacts, artifact_url, checksums, signed_by, sbom) - PK: (repo_url, hash, branch)Review extension:
review_items(repo_url, hash, branch, type, state, base, base_tip, head, head_tip, depends_on, closes, reviewers, pull_request_*, commit_ref, file, old_line, new_line, old_line_end, new_line_end, review_state, suggestion) - PK: (repo_url, hash, branch)review_branch_observations(repo_url, branch, tip, branch_exists, observed_at) - PK: (repo_url, branch) — transient cache of the live remote tip for every branch any open PR's head or base points at, across the workspace and registered forks. Refreshed by RefreshOpenPRBranches after fetch; consumed by the head-advanced / head-deleted / base-advanced / base-deleted notifications.Versioning: core_commits.edits stores raw header value; core_commits_version is authoritative. Use cache.ResolveToCanonical() / cache.GetLatestVersion().
Cross-repo proposals: edit resolution is gated to same-repo edits (GITMSG.md §1.5), so a cross-repo edit (e.g. a fork editing your issue) is an inert proposal until the owner acts. proposals.Accept applies it as the owner's own same-repo mirror edit carrying accepts=<proposal>, which wins resolution and, on processing, derives core_edit_acceptances; the proposer learns via that mirror on the gitmsg data branch, so acceptance needs no published marker. proposals.Decline publishes a durable marker at refs/gitmsg/core/declines/* so the proposer learns and the owner's choice survives a re-clone. Both clear the proposer's ✎ marker; accept takes precedence over decline.
core_commits carries effective_* generated columns (effective_message, effective_author_name, effective_author_email, effective_timestamp) that COALESCE the latest edit's content (resolved_message) and origin-author/origin-time (set on imported content) over the raw fields. Each extension has a *_items_resolved view that joins its tables onto core_commits and projects the generated columns under the legacy display names:
CREATE VIEW {ext}_items_resolved AS
SELECT
c.effective_message AS resolved_message,
c.effective_author_name AS author_name,
c.effective_timestamp AS timestamp,
...,
COALESCE(e.type, 'default') as type, e.field1, ...
FROM core_commits c
LEFT JOIN {ext}_items e ON c.repo_url = e.repo_url AND c.hash = e.hash AND c.branch = e.branch;
This ensures items are found regardless of whether they have extension-specific records. The denormalized resolved-state columns (resolved_message, has_edits, is_retracted) are written exclusively by applyEditToCanonical (core/cache/versions.go).
When to bypass the view: the *_items_resolved views are right for typical list/show queries where the WHERE clause is on core_commits columns (timestamp, repourl, etc.) and the result needs every commit-as-an-item. Bypass them — JOIN `corecommits` directly to the extension table — when:
pm_items.state = 'open', social_items.original_*). Driving from the small extension table avoids a planner mishap where core_commits (millions of rows) becomes the outer table.social_items.reply_to_*).Examples already in the codebase: social.GetThread (recursive CTE on social_items), social.GetNotifications (drives from social_items joined to core_commits). Both bypass the resolved view because the view forced a full scan on a 1M-commit cache.
Ref format: [repo_url]#type:value
https://github.com/user/repo#commit:abc123def456 - full ref#commit:abc123def456 - workspace-relative refcommit, branch, tag, file, listVirtual commits: Referenced in GitMsg-Ref but not yet fetched. Stored with is_virtual = 1 and full metadata. When fetched, is_virtual flips to 0.
Workspace refs (refs/gitmsg/*): extension data branches (gitmsg/<ext>) and these classes of state refs:
refs/gitmsg/<ext>/config — per-extension JSON config (single ref)refs/gitmsg/core/forks/<urlHash> — one ref per registered fork (per-element layout, no shared write target — concurrent fork adds across clones don't collide)refs/gitmsg/core/declines/<hash> — one ref per declined cross-repo proposal (subject = the proposal ref); published so the proposer's ✎ marker clears on their next fetch and the owner's decline survives a re-clone (acceptance needs no marker: it rides the owner's mirror edit)refs/gitmsg/<ext>/lists/<name>/_meta + .../items/<refHash> — list metadata at _meta, members as per-element refs (same rationale; metadata lives under _meta because git refuses to create child refs while a same-named parent ref exists)| Repo Type | Cache (core_commits) | Storage (repositories/) |
|---|---|---|
| Workspace | Full history, all branches (*) | N/A (uses workdir) |
Followed (*) | Full history, all branches | Persistent |
| Followed (specific branch) | Full history, incremental | Persistent |
| Non-followed | 30-day window | Can be deleted anytime |
All-branch following (branch = "*"): Commits are stored with their actual git refname (e.g., main, gitmsg/social, feature/x). The workspace always uses all-branch semantics. Deduplication and stale marking operate at the repo level via FilterUnfetchedCommitsByRepo / MarkCommitsStaleByRepo.
Switching modes: cache.ResetRepositoryData() clears old commits and extension items when switching between specific branch and *. Next fetch rebuilds with correct branches.
{extension_name}_ prefix(repo_url, hash) composite FK to core_commitscache.ExecLocked/QueryLocked for DB accessstorage.GetStorageDir() hashes URL only; same URL with different branches shares storagemeta.HasCommits before using timestamps (zero-value edge case)Any S3-compatible bucket (AWS S3, Cloudflare R2, DigitalOcean Spaces, MinIO, etc.) can be a git remote via the s3:// remote helper in core/objstore (per GITMSG.md §1.3). The only stored URL shape is s3://<endpoint>/<bucket>/<prefix>. gitsocial push (or an explicit gitsocial push --site-only) also uploads a browser-only static site alongside the repo, served straight from the bucket layout.
Two docs split by surface:
Cobra-generated — run gitsocial --help or gitsocial <group> --help for the authoritative, current list.
status, fetch, config, settings, log, search, show, explore, history, notifications, fork, id, tuiimport {all,pm,release,review,social}social, pm, release, review, memo — each adds status/config + its own verbs (and init, except memo, which inits per-tier)Planned extensions: cicd, ops, security, dm, portfolio
Two-panel layout using Bubbletea: Nav (left) + Content (right). See documentation/TUI-KEYS.md for key bindings.
┌─ Navigation ────────────┐┌─ Content ────────────────────────────────┐
│ Search ││ │
│ Notifications (3) ││ Timeline / Post / Repository / Search │
│ ─────────────────────── ││ │
│ ▸ Social ││ View content based on selection │
│ PM ││ │
│ ─────────────────────── ││ │
│ Settings ││ │
├─────────────────────────┤├──────────────────────────────────────────┤
│ Current dir ││ Context-sensitive keybindings │
└─────────────────────────┘└──────────────────────────────────────────┘
library/tui/
├── app.go / host.go # main tea.Model + view dispatch / shared state
├── tuicore/ # infrastructure + core views (view_/component_/registry_/util_/bus)
├── tuisocial/ # social views
├── tuipm/ # PM views
├── tuirelease/ # release views
├── tuireview/ # review views
├── tuimemo/ # memo views
└── test/ # headless integration tests (see TUI-TESTS.md)
| Prefix | Purpose | Examples |
|---|---|---|
view_ | Routable views | view_timeline.go, view_issues.go |
component_ | Reusable stateful components | component_nav_panel.go |
registry_ | Global registries | registry_nav.go |
form_ | Modal form overlays | form_issue.go |
version_item_ | History-picker version items (hero-card detail render) | version_item_issue.go |
util_ | Stateless utilities | util_render.go, util_keys.go |
tui/tuiXX/ directoryview_*.go filesutil_register.go with Register(host) functionapp.goIf more ceremony needed, we over-engineered.