documentation/ARCHITECTURE.md · main · 2026-09-06

GitSocial Architecture

Single Go library with thin clients: CLI/TUI (direct) and JSON-RPC (stdio).

DevelopmentDirectory StructurePackage ReferenceCache ArchitectureS3 Remote BackendCLI CommandsTUI


Development

Branching & Builds

Trunk-based: main is the integration branch and stays clean. Develop each feature on a feature/<name> branch; rebase on main and fast-forward merge so history stays linear.

git switch main && git pull --ff-only        # refresh before branching and after any merge
git switch -c feature/<name>                 # start a feature
git fetch origin && git rebase main          # keep current with main
git switch main && git merge --ff-only feature/<name>   # integrate
git branch -d feature/<name>                 # clean up

Build, test & run:

go build -o bin/gitsocial ./cli/gitsocial     # Build CLI
go build -o bin/ ./...                        # Compile-check everything (all mains land in bin/, never the repo root)
go test ./...                                 # Run tests
bin/gitsocial social timeline                 # Run command
bin/gitsocial tui                             # Launch TUI

Test & Lint

The gate is scripts/check.sh, in two tiers. scripts/check.sh --quick runs the prose check, go vet, golangci-lint and every test except the guarded ones; the pre-push hook runs it on every push (about 90 s warm when every package reruns, less after a change in one package). scripts/check.sh runs the same with GITSOCIAL_TEST_FULL=1, so the guarded tests run too (about 4 min warm); run it before merging to main and at release. Stages run in order, the failing stage is named, and any failure exits non-zero. A missing golangci-lint fails unless --skip-lint is passed.

Guarded tests call fullTierOnly: the TUI matrices TestSmoke, TestSequence and TestGolden/LayoutProperties, the CLI --json walk TestCommandTreeJSONOutput, and the TestS3Helper_* child-process tests. -race and the browser site battery run at release (scripts/release.sh preflight). -short skips 66 real-git subtests and is a local smoke run, never a tier.

scripts/prose-check.sh is stage 0. It counts STYLE.md violations (em-dashes, comment blocks over three lines, long Short and flag help) and fails when a count rises above scripts/prose-baseline.txt; --update accepts the current counts, --list <rule> prints the offending lines. Commit subjects over 72 characters in the pushed range fail outright.

scripts/check.sh --quick         # the push tier
scripts/check.sh                 # the full tier
scripts/check.sh --skip-lint     # golangci-lint not installed (not a gate run)
scripts/check.sh -short ./...    # extra args go to the test stage (smoke run, not a gate run)
git config core.hooksPath scripts/hooks   # install the pre-push hook (once per clone)
GITSOCIAL_SKIP_GATE=1 git push   # skip the gate once

Coverage (scripts/coverage.sh, writes .test-artifacts/coverage/): runs go test ./... -coverpkg=./... so code exercised by another package's integration tests gets credit, then reports the statement-weighted total (not a mean of per-package percentages), a ranked per-package table including packages with no test files of their own, and every function at 0.0%. The total is a floor: coverage cannot see the S3 helper tests (they run the helper as a child process, so helper_push.go / thin.go look untested) or the ~40 browser suites behind -tags sitetest (site_pages*.go).

go test ./...                    # All tests
go test ./library/core/cache     # Specific package
go test -v ./...                 # Verbose output
go test -race ./...              # Race detector
go test -cover ./...             # Coverage summary
golangci-lint run --fix ./...    # Lint & fix code

scripts/test.sh                  # Streams per-test progress; wraps `go test -json` (accepts the same args, e.g. `scripts/test.sh -race ./...`)

go test ./library/tui/test/...   # Headless TUI suite (smoke/display/golden/nav/sequence; see TUI-TESTS.md)

go test -tags sitetest -timeout 30m ./library/core/objstore/   # Static-site browser suites (wraps scripts/site-test.sh; needs node; see STATIC-SITE.md)

Code Rules

Layer Dependencies

library/extensions/* → library/core/* → stdlib only
          ↓                 ↓
  cli/gitsocial/        (no circular refs)
  library/tui/
  library/rpc/
  library/import/  → library/extensions/* + library/core/protocol

Do

Never

Code Patterns

Error Handling by Layer

Core packages (cache, git, protocol)  → return error (idiomatic Go)
Extension public API (social.*)       → return Result[T] (user-facing codes)
Internal helpers                      → return error

Rules:

  1. Always wrap errors: fmt.Errorf("operation: %w", err)
  2. At API boundaries, convert to Result[T] for user-friendly error codes
  3. For batch operations that continue on failure, log instead of returning
  4. Intentional suppressions must be commented

Common patterns (see code for examples)


Directory Structure

gitsocial/                     # module github.com/gitsocial-org/gitsocial
├── cli/gitsocial/             # CLI thin client; builds the binary
├── library/                   # Go library — single source of truth
│   ├── core/                  # Shared infrastructure
│   │   ├── git/               # Git operations
│   │   ├── protocol/          # GitMsg protocol parsing
│   │   ├── gitmsg/            # Protocol-level storage
│   │   ├── cache/             # SQLite operations
│   │   ├── storage/           # Bare repo management
│   │   ├── objstore/          # S3 remote backend (stdlib client + git remote helper)
│   │   ├── fetch/             # Fetch orchestration + processing
│   │   ├── identity/          # Identity declarations + verification
│   │   │   └── forge/         # Forge adapters (GitHub, …)
│   │   ├── notifications/     # Notification aggregation
│   │   ├── search/            # Cross-extension search
│   │   ├── settings/          # User settings + config paths
│   │   ├── log/               # Structured logging
│   │   ├── text/              # String helpers
│   │   └── result/            # Result[T] type
│   ├── extensions/
│   │   ├── social/            # Posts, lists, timeline
│   │   ├── pm/                # Issues, milestones, sprints
│   │   ├── release/           # Releases, versions, artifacts
│   │   ├── review/            # Pull requests, code reviews
│   │   └── memo/              # Tiered memos (knowledge as commits)
│   ├── proposals/             # Cross-repo proposals (gate; accept + decline engine)
│   ├── import/                # Platform import pipeline
│   │   ├── github/            # GitHub adapter (gh CLI)
│   │   └── gitlab/            # GitLab adapter
│   ├── clientfetch/           # Thin-client fetch orchestration (CLI/TUI)
│   ├── rpc/                   # JSON-RPC server (stdio) — thin-client surface
│   └── tui/                   # TUI views — thin client
├── documentation/             # Protocol + architecture docs
├── scripts/                   # Build/release/test scripts (release.sh, install.sh, test.sh, site-test.sh)
└── specs/                     # Protocol specifications

Outside the repo tree:

~/.config/gitsocial/           # User config; honors `XDG_CONFIG_HOME`
├── settings.json              # Machine-specific settings
└── personal/                  # Personal-tier bare repo (override: `GITSOCIAL_PERSONAL_REPO`)

~/.cache/gitsocial/            # Cache dir, `--cache-dir` overrides
├── cache.db                   # SQLite (commits + extension tables)
├── repositories/              # Bare git clones (cleaned up periodically)
├── forks/                     # Fork bare clones
├── imports/                   # Import mapping files (per repo URL slug)
└── memo/session/              # Per-session memo bare repos

Package Reference

PackageKey TypesKey Exports
core/git
Git operations
Commit, FileDiff, Hunk, DiffLine, DiffStatsGetCommits, CreateCommit, ReadRef, WriteRef, GetDiff, GetFileDiff, GetFileContent, GetDiffStats, MergeBranches, SquashMerge, RebaseMerge, ForceMerge, RebaseBranch, RangeDiff, PatchesEqual, GetBehindCount, GetMergeBase, GetUserName, GetGitConfig, CreateSignedCommitTree, VerifyCommitSignature, GetCommitSignerKey
core/protocol
Message parsing
Header, Message, Origin, TrailerParseMessage, ParseHeader, CreateHeader, FormatMessage, ParseRef, CreateRef, FormatShortRef, QuoteContent, ApplyOrigin, ExtractTrailers, Trailer, IsClosingTrailer
core/cache
SQLite operations
Repository, Commit, TrailerRefOpen, DB, ExecLocked, QueryLocked, InsertCommits, FilterUnfetchedCommitsByRepo, MarkCommitsStaleByRepo, ResetRepositoryData, RegisterMigration, ToNullString, ToNullInt64, GetTrailerRefsTo, TrailerRef
core/gitmsg
Protocol-level storage
ResolveRepoURL, Push, ReadExtConfig, WriteList, GetHistory, GetExtBranch, IsExtInitialized, GetForks, AddFork, AddForks, RemoveFork
core/storage
Bare repo management
EnsureRepository, GetStorageDir, FetchRepository
core/objstore
S3 remote backend (see S3 Remote Backend)
Client, Config, Capability, HelperEnvNewClient, ParseS3URL, RunHelper, HelperEnvFromOS, ListRemoteRefs, PushSite, PushArtifactObjects, PutObjectToRemote
core/fetch
Fetch orchestration
FetchAll, FetchRepository, FetchForks, CommitProcessor, PostFetchHook
core/settings
User settings
Get, Set, ListAll
core/search
Cross-extension search
Search, Params, Result, Item, Group, GroupedItem, FormatResult, IsValidGroupBy
core/result
Result type
Result[T], ErrorOk, Err, ErrWithDetails
core/notifications
Notification aggregation
Notification, Provider, FilterRegisterProvider, GetAll, GetUnreadCount, MarkAsRead, MarkAsUnread, MarkAllAsRead, MarkAllAsUnread, MentionProcessor, ExtractMentions, TrailerProcessor
core/identity
Identity verification
Identity, ResolvedIdentity, DNSIdentity, Binding, Source, VerifyCandidateVerifyBinding, IsVerified, IsVerifiedCommit, LookupBinding, VerifyCandidates, NormalizeSignerKey, NormalizeEmail, ResolveIdentity
core/identity/forge
Forge adapters for identity verification
Forge, GPGKey, CommitVerificationForge, Register, Lookup, LookupForRepo, ParseRepoURL, NewGitHub, GPGKey, CommitVerification
extensions/social
Social layer
Post, SocialItemGetPosts, CreatePost, GetTimeline, Fetch
extensions/pm
Project management
Issue, Milestone, Sprint, PMNotificationGetIssues, CreateIssue, GetMilestones, GetSprints, MessageToPMItem, FetchRepository, Processors
extensions/release
Release management
Release, ReleaseItem, ReleaseNotificationCreateRelease, EditRelease, GetReleases, GetSingleRelease, MessageToReleaseItem, FetchRepository, Processors
extensions/review
Code review
PullRequest, Feedback, ReviewSummary, StackEntry, ReviewNotificationCreatePR, GetPR, UpdatePR, MergePR, ClosePR, RetractPR, MarkReady, ConvertToDraft, UpdatePRTips, SyncPRBranch, GetPRVersions, ComparePRVersions, GetVersionAwareReviews, CreateFeedback, GetReviewSummary, MessageToReviewItem, FetchRepository, GetPullRequestsWithForks, GetStack, GetDependents, Processors
extensions/memo
Tiered memos (knowledge as commits)
Memo, MemoItem, Tier, SessionInfoCreateMemo, EditMemo, RetractMemo, PromoteMemo, ListMemos, GetSingleMemo, InitProject, InitPersonal, InitSession, ListSessions, GCSession, PushPersonal, FetchPersonal, PushSession, FetchSession, SyncAllTierReposToCache, AddInherit, RemoveInherit, ListInherits, IsInherited
proposals
Cross-repo proposals
OutcomeAccept, Decline
import
Platform import pipeline
SourceAdapter, ImportPlan, Stats, MappingFileRun, SourceAdapter, ReadMapping, WriteMapping, MappingKey, ResolveHost, MapLabels
import/github
GitHub adapter
New, CheckGH, Adapter.FetchPM, Adapter.FetchReleases, Adapter.FetchReview, Adapter.FetchSocial

Terminology

TermContextMeaning
originalGITSOCIAL fieldPost being commented/reposted/quoted
canonicalVersioningFirst version of a message (before edits)
editsGITMSG fieldReference to canonical version being edited

Cache Architecture

Key principle: Storage (repositories/) can be deleted anytime. Fetch strategy is determined by cache.db metadata, not storage state.

Staleness: The cache is append-only, but commits that no longer exist in their source branch (e.g., after rebase or force-push) are marked with a stale_since timestamp via cache.MarkCommitsStale() (single-branch) or cache.MarkCommitsStaleByRepo() (all-branch). Stale commits are excluded from timeline and list queries but remain visible (dimmed) in thread and detail views to preserve discussion context.

SQLite tuning: WAL mode, 64MB cache (_cache_size=-65536), memory temp store, 16 max connections, 256MB mmap.

Schema

Core tables:

Social extension:

Release extension:

Review extension:

Versioning: core_commits.edits stores raw header value; core_commits_version is authoritative. Use cache.ResolveToCanonical() / cache.GetLatestVersion().

Cross-repo proposals: edit resolution is gated to same-repo edits (GITMSG.md §1.5), so a cross-repo edit (e.g. a fork editing your issue) is an inert proposal until the owner acts. proposals.Accept applies it as the owner's own same-repo mirror edit carrying accepts=<proposal>, which wins resolution and, on processing, derives core_edit_acceptances; the proposer learns via that mirror on the gitmsg data branch, so acceptance needs no published marker. proposals.Decline publishes a durable marker at refs/gitmsg/core/declines/* so the proposer learns and the owner's choice survives a re-clone. Both clear the proposer's ✎ marker; accept takes precedence over decline.

Resolved Views

core_commits carries effective_* generated columns (effective_message, effective_author_name, effective_author_email, effective_timestamp) that COALESCE the latest edit's content (resolved_message) and origin-author/origin-time (set on imported content) over the raw fields. Each extension has a *_items_resolved view that joins its tables onto core_commits and projects the generated columns under the legacy display names:

CREATE VIEW {ext}_items_resolved AS
SELECT
    c.effective_message AS resolved_message,
    c.effective_author_name AS author_name,
    c.effective_timestamp AS timestamp,
    ...,
    COALESCE(e.type, 'default') as type, e.field1, ...
FROM core_commits c
LEFT JOIN {ext}_items e ON c.repo_url = e.repo_url AND c.hash = e.hash AND c.branch = e.branch;

This ensures items are found regardless of whether they have extension-specific records. The denormalized resolved-state columns (resolved_message, has_edits, is_retracted) are written exclusively by applyEditToCanonical (core/cache/versions.go).

When to bypass the view: the *_items_resolved views are right for typical list/show queries where the WHERE clause is on core_commits columns (timestamp, repourl, etc.) and the result needs every commit-as-an-item. Bypass them — JOIN `corecommits` directly to the extension table — when:

Examples already in the codebase: social.GetThread (recursive CTE on social_items), social.GetNotifications (drives from social_items joined to core_commits). Both bypass the resolved view because the view forced a full scan on a 1M-commit cache.

Refs and Keys

Ref format: [repo_url]#type:value

Virtual commits: Referenced in GitMsg-Ref but not yet fetched. Stored with is_virtual = 1 and full metadata. When fetched, is_virtual flips to 0.

Workspace refs (refs/gitmsg/*): extension data branches (gitmsg/<ext>) and these classes of state refs:

Fetch Rules

Repo TypeCache (core_commits)Storage (repositories/)
WorkspaceFull history, all branches (*)N/A (uses workdir)
Followed (*)Full history, all branchesPersistent
Followed (specific branch)Full history, incrementalPersistent
Non-followed30-day windowCan be deleted anytime

All-branch following (branch = "*"): Commits are stored with their actual git refname (e.g., main, gitmsg/social, feature/x). The workspace always uses all-branch semantics. Deduplication and stale marking operate at the repo level via FilterUnfetchedCommitsByRepo / MarkCommitsStaleByRepo.

Switching modes: cache.ResetRepositoryData() clears old commits and extension items when switching between specific branch and *. Next fetch rebuilds with correct branches.

Extension Guidelines

Known Limitations

  1. storage.GetStorageDir() hashes URL only; same URL with different branches shares storage
  2. Check meta.HasCommits before using timestamps (zero-value edge case)

S3 Remote Backend

Any S3-compatible bucket (AWS S3, Cloudflare R2, DigitalOcean Spaces, MinIO, etc.) can be a git remote via the s3:// remote helper in core/objstore (per GITMSG.md §1.3). The only stored URL shape is s3://<endpoint>/<bucket>/<prefix>. gitsocial push (or an explicit gitsocial push --site-only) also uploads a browser-only static site alongside the repo, served straight from the bucket layout.

Two docs split by surface:


CLI Commands

Cobra-generated — run gitsocial --help or gitsocial <group> --help for the authoritative, current list.

Planned extensions: cicd, ops, security, dm, portfolio


TUI

Two-panel layout using Bubbletea: Nav (left) + Content (right). See documentation/TUI-KEYS.md for key bindings.

┌─ Navigation ────────────┐┌─ Content ────────────────────────────────┐
│   Search                ││                                          │
│   Notifications (3)     ││  Timeline / Post / Repository / Search   │
│ ─────────────────────── ││                                          │
│ ▸ Social                ││  View content based on selection         │
│   PM                    ││                                          │
│ ─────────────────────── ││                                          │
│   Settings              ││                                          │
├─────────────────────────┤├──────────────────────────────────────────┤
│ Current dir             ││ Context-sensitive keybindings            │
└─────────────────────────┘└──────────────────────────────────────────┘

Structure

library/tui/
├── app.go / host.go     # main tea.Model + view dispatch / shared state
├── tuicore/             # infrastructure + core views (view_/component_/registry_/util_/bus)
├── tuisocial/           # social views
├── tuipm/               # PM views
├── tuirelease/          # release views
├── tuireview/           # review views
├── tuimemo/             # memo views
└── test/                # headless integration tests (see TUI-TESTS.md)

File Naming Convention

PrefixPurposeExamples
view_Routable viewsview_timeline.go, view_issues.go
component_Reusable stateful componentscomponent_nav_panel.go
registry_Global registriesregistry_nav.go
form_Modal form overlaysform_issue.go
version_item_History-picker version items (hero-card detail render)version_item_issue.go
util_Stateless utilitiesutil_render.go, util_keys.go

Adding a New Extension

  1. Create tui/tuiXX/ directory
  2. Add views as view_*.go files
  3. Add util_register.go with Register(host) function
  4. Call from app.go

If more ceremony needed, we over-engineered.