Content Pipeline Architecture

NOTE

This document details the Content Pipeline Architecture specified and enhanced in PR #265 (feat/ui-downloads), linking discovery, universal parsing, resolution, delivery, and post-extraction manifest factories with the Unified Downloads Browser.

The GenHub content system uses a three-tier pipeline architecture that transforms external content sources into installable content with full manifest and CAS (Content-Addressable Storage) integration.

Pipeline Overview

Tier 1: ContentOrchestrator

Location: GenHub.Core/Interfaces/Content/IContentOrchestrator.cs

The orchestrator is the system-wide coordinator for all content operations.

Responsibilities

Operation	Method	Description
Search	`SearchAsync()`	Broadcasts query to all providers, aggregates results
Acquire	`AcquireContentAsync()`	Downloads, extracts, stores, and registers content
Cache	`IDynamicContentCache`	System-wide caching for performance

Search Flow

csharp

// User initiates search in DownloadsBrowserView
var results = await _orchestrator.SearchAsync(new ContentSearchQuery
{
    SearchTerm = "Rise of the Reds",
    ContentType = ContentType.Mod,
    TargetGame = GameType.ZeroHour
});

Tier 2: Content Providers

Base: GenHub/Features/Content/Services/ContentProviders/BaseContentProvider.cs

Providers are source-specific facades that orchestrate the internal pipeline.

Provider Pattern

csharp

public abstract class BaseContentProvider : IContentProvider
{
    protected abstract IContentDiscoverer Discoverer { get; }
    protected abstract IContentResolver Resolver { get; }
    protected abstract IContentDeliverer Deliverer { get; }

    // Common pipeline orchestration
    public virtual async Task<OperationResult<IEnumerable<ContentSearchResult>>> SearchAsync(
        ContentSearchQuery query, CancellationToken cancellationToken = default)
    {
        var providerDefinition = GetProviderDefinition();
        return await Discoverer.DiscoverAsync(providerDefinition, query, cancellationToken);
    }
}

Registered Providers

Provider	Discoverer	Parser	Notes
ModDB	`ModDBDiscoverer`	`ModDBPageParser` (AngleSharp)	Uses Playwright for WAF bypass
CNC Labs	`CNCLabsMapDiscoverer`	AngleSharp HTML	Direct HTTP scraping
AOD Maps	`AODMapsDiscoverer`	`AODMapsPageParser`	Pagination support
Community Outpost	`CommunityOutpostDiscoverer`	`GenPatcherDatCatalogParser`	`.dat` catalog format
GitHub	`GitHubDiscoverer`	GitHub API JSON	Release assets
Generals Online	`GeneralsOnlineDiscoverer`	GitHub API	Multi-variant releases
File System	`FileSystemDiscoverer`	Direct scan	Local manifests

Tier 3: Pipeline Components

Discoverers (`IContentDiscoverer`)

Location: GenHub.Core/Interfaces/Content/IContentDiscoverer.cs

Discoverers fetch catalog data from external sources and delegate to parsers.

csharp

public interface IContentDiscoverer : IContentSource
{
    Task<OperationResult<ContentDiscoveryResult>> DiscoverAsync(
        ContentSearchQuery query,
        CancellationToken cancellationToken = default);

    // Overload with provider definition for data-driven configuration
    Task<OperationResult<ContentDiscoveryResult>> DiscoverAsync(
        ProviderDefinition? provider,
        ContentSearchQuery query,
        CancellationToken cancellationToken = default);
}

Key principle: Discoverers handle network concerns (timeouts, retries, WAF bypass) but do NOT parse data themselves—that's the parser's job.

Parsers (`ICatalogParser`, `IWebPageParser`)

Locations:

GenHub.Core/Interfaces/Providers/ICatalogParser.cs
GenHub.Core/Interfaces/Parsers/IWebPageParser.cs

Parsers transform raw data (HTML, JSON, .dat files) into ContentSearchResult objects.

Parser	Format	Source
`GenPatcherDatCatalogParser`	`.dat` pipe-delimited	Community Outpost
`ModDBPageParser`	HTML	ModDB
`AODMapsPageParser`	HTML	AOD Maps
AngleSharp	Generic HTML	CNC Labs

Resolvers (`IContentResolver`)

Location: GenHub.Core/Interfaces/Content/IContentResolver.cs

Resolvers transform lightweight search results into complete ContentManifest blueprints.

csharp

public interface IContentResolver
{
    string ResolverId { get; }

    Task<OperationResult<ContentManifest>> ResolveAsync(
        ContentSearchResult discoveredItem,
        CancellationToken cancellationToken = default);

    Task<OperationResult<ContentManifest>> ResolveAsync(
        ProviderDefinition? provider,
        ContentSearchResult discoveredItem,
        CancellationToken cancellationToken = default);
}

Resolution tasks:

Fetch detail page for full metadata (description, screenshots)
Extract download URL
Determine target game and content type
Build manifest structure

Deliverers (`IContentDeliverer`)

Location: GenHub.Core/Interfaces/Content/IContentDeliverer.cs

Deliverers download content files and prepare them for storage.

csharp

public interface IContentDeliverer : IContentSource
{
    bool CanDeliver(ContentManifest manifest);

    Task<OperationResult<ContentManifest>> DeliverContentAsync(
        ContentManifest packageManifest,
        string targetDirectory,
        IProgress<ContentAcquisitionProgress>? progress = null,
        CancellationToken cancellationToken = default);

    Task<OperationResult<bool>> ValidateContentAsync(
        ContentManifest manifest,
        CancellationToken cancellationToken = default);
}

CAS-Resident Content (Skip Delivery)

Publisher manifest factories like AODMapsManifestFactory and CNCLabsManifestFactory download the file into CAS during resolution and register it as a ContentAddressable file with a hash but no DownloadUrl. (In contrast, ModDBManifestFactory registers files with RemoteDownload and DownloadUrl via AddRemoteFileAsync, leaving acquisition to the deliverer stage). The HTTP deliverer cannot handle such files (CanDeliver requires an http download URL), so the matching content providers short-circuit preparation: when every file is already ContentAddressable with a hash, the provider returns the manifest as-is and the orchestrator's delivery stage is skipped. ContentValidator and ContentStorageService both resolve such files against CAS by hash (not the staging folder), so the file is found end-to-end. A pool-agnostic hash lookup is used as a fallback so files stored under one content-type pool are still found when the manifest reports another.

Manifest Factories (`IContentManifestFactory`)

Location: GenHub.Core/Interfaces/Manifest/IContentManifestFactory.cs

Factories create proper ContentManifest objects after downloading, handling publisher-specific logic.

Factory	Publisher	Features
`ModDBManifestFactory`	ModDB	ID format: `1.YYYYMMDD.moddb-{author}.{type}.{name}`
`CNCLabsManifestFactory`	CNC Labs	Map-specific metadata
`AODMapsManifestFactory`	AOD Maps	Referer header handling
`GitHubManifestFactory`	GitHub	Release asset handling
`SuperHackersManifestFactory`	The Super Hackers	Multi-game releases (Generals + ZH)

Archive Handling

The ContentManifestBuilder.AddDownloadedFileAsync() method automatically handles archives:

Supported formats: ZIP, RAR, 7z, TAR, GZ (via SharpCompress library)

Detection: By file signature (magic bytes), NOT file extension

ContentManifest Builder

Location: GenHub/Features/Manifest/ContentManifestBuilder.cs

The fluent builder API for manifest creation:

csharp

var manifest = manifestBuilder
    .WithBasicInfo(publisherId, contentName, manifestVersion)
    .WithContentType(ContentType.Mod, GameType.ZeroHour)
    .WithPublisher(
        name: "ModDB - Author Name",
        website: "https://moddb.com",
        publisherType: "moddb")
    .WithMetadata(
        description: details.Description,
        tags: ["mod", "zerohour"],
        iconUrl: details.PreviewImage)
    .Build();

// Add downloaded file (handles archive extraction automatically)
await manifest.AddDownloadedFileAsync(
    relativePath: "content.zip",
    downloadUrl: "https://example.com/download",
    refererUrl: detailPageUrl,  // For sites requiring referer
    userAgent: customUserAgent); // Triggers Playwright if set

Key Methods

Method	Purpose
`AddDownloadedFileAsync()`	Downloads, extracts archives, stores in CAS
`AddFilesFromDirectoryAsync()`	Scans directory, hashes files, adds to manifest
`AddLocalFileAsync()`	Adds existing local file
`AddContentAddressableFileAsync()`	Adds CAS reference by hash
`AddDependency()`	Adds content dependency

Manifest ID System

Documentation: manifest-id-system.md

IDs follow a deterministic format:

{version}.{userVersion}.{publisherId}.{contentType}.{contentName}

Examples:

1.20190826.moddbhan.mod.hanpatchv32 - ModDB mod
1.104.ea.gameinstallation.zerohour - Base game

Downloads View Integration

User Flow

Acquisition Flow

Per-Publisher Implementation Checklist

To add support for a new publisher:

1. Create Constants

csharp

// GenHub.Core/Constants/MyPublisherConstants.cs
public static class MyPublisherConstants
{
    public const string PublisherPrefix = "mypub";
    public const string PublisherName = "My Publisher";
    public const string PublisherWebsite = "https://mypub.example.com";
}

2. Create Discoverer

csharp

public class MyPublisherDiscoverer : IContentDiscoverer
{
    public async Task<OperationResult<ContentDiscoveryResult>> DiscoverAsync(
        ContentSearchQuery query, CancellationToken ct)
    {
        // 1. Fetch catalog from source
        // 2. Parse into ContentSearchResult objects
        // 3. Apply query filters
        return OperationResult<ContentDiscoveryResult>.CreateSuccess(
            new ContentDiscoveryResult { Items = results });
    }
}

3. Create Resolver (if needed)

csharp

public class MyPublisherResolver : IContentResolver
{
    public async Task<OperationResult<ContentManifest>> ResolveAsync(
        ContentSearchResult item, CancellationToken ct)
    {
        // Fetch detail page, build full manifest
    }
}

4. Create Manifest Factory

csharp

public class MyPublisherManifestFactory : IContentManifestFactory
{
    public bool CanHandle(ContentManifest manifest) =>
        manifest.Publisher.Name.Contains("My Publisher");

    public async Task<ContentManifest> CreateManifestAsync(...)
    {
        // Build manifest with AddDownloadedFileAsync()
    }
}

5. Register in DI

csharp

// ContentPipelineModule.cs
services.AddTransient<IContentDiscoverer, MyPublisherDiscoverer>();
services.AddTransient<IContentManifestFactory, MyPublisherManifestFactory>();

Publisher Configuration - Data-driven publisher settings
Discovery Flow - Visual discovery workflow
Manifest ID System - ID generation rules
Publisher Infrastructure - Clean architecture for content publishers

Content Pipeline Architecture ​

Pipeline Overview ​

Tier 1: ContentOrchestrator ​

Responsibilities ​

Search Flow ​

Tier 2: Content Providers ​

Provider Pattern ​

Registered Providers ​

Tier 3: Pipeline Components ​

Discoverers (IContentDiscoverer) ​

Parsers (ICatalogParser, IWebPageParser) ​

Resolvers (IContentResolver) ​

Deliverers (IContentDeliverer) ​

CAS-Resident Content (Skip Delivery) ​

Manifest Factories (IContentManifestFactory) ​

Archive Handling ​

ContentManifest Builder ​

Key Methods ​

Manifest ID System ​

Downloads View Integration ​

User Flow ​

Acquisition Flow ​

Per-Publisher Implementation Checklist ​

1. Create Constants ​

2. Create Discoverer ​

3. Create Resolver (if needed) ​

4. Create Manifest Factory ​

5. Register in DI ​

Related Documentation ​

Content Pipeline Architecture

Pipeline Overview

Tier 1: ContentOrchestrator

Responsibilities

Search Flow

Tier 2: Content Providers

Provider Pattern

Registered Providers

Tier 3: Pipeline Components

Discoverers (`IContentDiscoverer`)

Parsers (`ICatalogParser`, `IWebPageParser`)

Resolvers (`IContentResolver`)

Deliverers (`IContentDeliverer`)

CAS-Resident Content (Skip Delivery)

Manifest Factories (`IContentManifestFactory`)

Archive Handling

ContentManifest Builder

Key Methods

Manifest ID System

Downloads View Integration

User Flow

Acquisition Flow

Per-Publisher Implementation Checklist

1. Create Constants

2. Create Discoverer

3. Create Resolver (if needed)

4. Create Manifest Factory

5. Register in DI

Related Documentation