It started with a friend mentioning it would be fun to run draft tournaments online. MTG Arena doesn't support private tournaments—you can play against each other, but organizing an actual draft pod with friends and running a bracket? Not happening.

So I figured: how hard could it be?

Around the same time, I'd been curious how well AI coding agents actually perform on complex projects. Claude Code had been getting a lot of praise, and this seemed like the perfect test case. Magic: The Gathering has over 28,000 unique cards, each able to interact with every other in ways the designers never anticipated. The comprehensive rules document is over 200 pages long.

I sat down and estimated the effort. Without AI assistance, building a complete platform—SDK, card sets, rules engine, game server, web frontend, deployment pipeline—would easily be a 6-month hobby project. Probably more, given the notorious complexity of MTG's rules. The layer system alone could eat a month.

A week and a half later, I had a working solution.

41,000 lines of Kotlin. 12,000 lines of TypeScript. A functional draft system. Real-time multiplayer. And it plays Magic—correctly. The layer system handles Humility. Triggered abilities stack in APNAP order. State-based actions cascade properly.

This was the moment it really hit me how fast this technology is moving.

What Vibe Coding Actually Looks Like

I barely wrote any code myself. Instead, I guided the process—describing what I wanted, reviewing what came back, course-correcting when the implementation drifted. Think of it as pair programming where your partner types 50x faster than you but needs you to keep the big picture in focus.

To keep Claude aligned with my vision, I set up a docs folder containing guidance on architecture decisions, coding style, and design principles. From these docs, I generated a CLAUDE.md file—a dedicated instruction set that Claude Code reads automatically—and kept it up-to-date as the project evolved. This upfront investment paid off continuously: Claude would follow established patterns without me having to repeat myself.

What I loved most was plan mode. Before Claude starts building anything significant, it asks clarifying questions: "How should this interact with existing triggers? Should the effect be optional? What happens if there are no valid targets?" Then it forms a concrete plan and waits for approval. This back-and-forth is invaluable. Half the bugs in software come from misunderstood requirements—plan mode surfaces those misunderstandings before any code gets written.

For architecture reviews, I'd occasionally pull in Gemini. With its million-token context window, I could dump the entire codebase into a single request and ask "does this structure make sense?" or "what am I missing?" The combination works well: Claude Code for the building, Gemini for the birds-eye sanity checks.

The result is a codebase I understand deeply—because I co-designed it—but didn't have to type out line by line. Would I have finished this project without AI assistance? Probably not. It would have joined the graveyard of ambitious side projects that never quite made it.

The Problem

The intuiting might be to take the obvious approach: model cards as class hierarchies. A Creature extends Permanent, which extends Card. Simple, intuitive, wrong.

The problem is that in Magic, a card's identity is fluid. That creature you control? An opponent can steal it. That instant in your hand? It might be a creature on the battlefield (thanks, Zoetic Cavern). That 2/2 bear? With the right enchantments, it's now a 7/7 flying artifact creature with lifelink.

Object-oriented inheritance can't handle this. You can't re-parent an object at runtime. So most engines end up with sprawling conditional logic, special cases, and inevitable bugs when two obscure cards interact in ways nobody tested.

Starting Simple

I started with Portal, the simplified beginner set from 1997. No instants (well a few, but they weren't called instants yet), few activated abilities, mostly just creatures and sorceries. A manageable scope to get the core engine working before tackling the full comprehensive rules.

There's something fitting about that. Portal was how I started playing Magic when I was 8 years old. Now, decades later, it became the foundation for building an engine that handles the full complexity of the game.

Once Portal worked—creatures attacking, sorceries resolving, life totals tracking—the architecture proved solid enough to layer on the rest. The stack. Triggered abilities. Replacement effects. The infamous layer system.

The Full Stack

The project splits into cleanly separated modules, each with a single responsibility:

The SDK

The SDK defines the vocabulary—what a card is, what effects exist, what costs mean. It's pure data structures and a DSL for defining cards. No game logic, no dependencies.

Think of it as the shared language that card definitions and the engine both speak. It provides the building blocks: mana costs, target filters, effect types, duration modifiers:

// The Effects facade - card definitions compose these primitives
Effects.DealDamage(3)                    // Bolt something
Effects.DrawCards(2)                     // Blue's favorite
Effects.GainLife(5)                      // White's thing
Effects.Destroy(EffectTarget.AnyTarget)  // Removal
Effects.ReturnToHand(target)             // Bounce
Effects.ModifyStats(4, 4, Duration.EndOfTurn)  // Pump spell

The SDK is deliberately minimal. It describes the "what" without the "how". A card has a mana cost, a type line, and abilities. An effect has a target and a duration. That's it. The actual game logic lives elsewhere.

The Card Sets

The card sets use that DSL to define actual cards. Portal, Alpha, custom sets—they're all just data. Each card is a declarative description of what it does:

val MonstrousGrowth = card("Monstrous Growth") {
    manaCost = "{1}{G}"
    typeLine = "Sorcery"

    spell {
        target = TargetCreature(filter = CreatureTargetFilter.Any)
        effect = ModifyStatsEffect(
            powerModifier = 4,
            toughnessModifier = 4,
            target = EffectTarget.ContextTarget(0)
        )
    }
}

Triggered abilities work the same way:

val ManOWar = card("Man-o'-War") {
    manaCost = "{2}{U}"
    typeLine = "Creature — Jellyfish"
    power = 2
    toughness = 2

    triggeredAbility {
        trigger = OnEnterBattlefield()
        target = TargetCreature()
        effect = ReturnToHandEffect(EffectTarget.ContextTarget(0))
    }
}

Traditional engines hardcode card logic. Want to add a new card? Write a new class, handle all its edge cases, hope you didn't break something else.

Argentum treats cards as pure data. The engine doesn't know about Monstrous Growth specifically. It knows how to create continuous effects. It knows how to handle targets. The card definition just connects these primitives. Adding new cards rarely requires engine changes.

Adding a new set means writing card definitions, nothing more. The sets don't know how the game works; they just describe what cards do.

The Rules Engine

The rules engine is the brain. Pure Kotlin, zero server dependencies. This is where the comprehensive rules live: the stack, priority system, combat phases, triggered abilities, state-based actions, replacement effects, and the layer system.

Entity-Component-System Architecture

Instead of class hierarchies, Argentum uses pure ECS. Everything is an entity—just a unique ID. Cards, players, spells on the stack, even continuous effects like "Giant Growth gave this +3/+3 until end of turn." All entities.

Each entity is defined by its components: pure data bags with no behavior. A creature on the battlefield might have components marking it as tapped, tracking its controller, counting its +1/+1 counters. The entity doesn't "know" it's tapped. It simply has—or doesn't have—that data attached.

All logic lives in systems. The action processor takes a game state and an action, then returns a new state plus events:

(GameState, GameAction) → ExecutionResult(GameState, List<GameEvent>)

The state is immutable. Every game action creates a fresh snapshot, which makes replay, undo, and network sync trivial. Completely deterministic: two identical inputs always produce identical outputs.

The Layer System

Here's where it gets interesting. When multiple effects modify a permanent, they apply in a specific order across seven layers:

1. Copy effects (Clone becomes a copy of target creature)
2. Control-changing effects (Mind Control steals a creature)
3. Text-changing effects (Magical Hack changes "swamp" to "island")
4. Type-changing effects (Blood Moon makes nonbasic lands Mountains)
5. Color-changing effects (Painter's Servant makes everything blue)
6. Ability-adding/removing effects (Glorious Anthem gives creatures +1/+1)
7. Power/toughness effects (with five sublayers: setting, modifying, counters, switching)

Get the order wrong and your engine produces incorrect game states.

Argentum separates "base state" from "projected state." The base state is what's stored—the permanent as it exists without modifications. The state projector calculates what players actually see by applying all active continuous effects in the correct layer order.

This separation is crucial. When an effect ends (Giant Growth wears off at end of turn), you don't need to "undo" anything—just remove the effect from the list and recalculate. The base state never changed.

The tricky part is dependencies. Sometimes effect A changes whether effect B applies, which might change whether effect A applies. Consider Humility (makes all creatures 1/1 with no abilities) and Opalescence (makes enchantments into creatures). If Humility is an enchantment... does it affect itself?

Argentum handles this with a trial application system—apply effects tentatively, detect when one effect's outcome depends on another's application, establish the dependency order, then apply for real. It's the kind of edge case that breaks most MTG engines. Argentum handles it correctly.

Because the engine is completely deterministic and has no I/O, testing is straightforward. Feed it a state and an action, verify the output. No mocking, no flakiness.

The Game Server

The game server wraps the engine for online play. It handles WebSocket connections, manages concurrent game sessions, authenticates players via JWT, and—critically—filters hidden information.

The engine produces a complete game state including everyone's hands and library order. The server ensures each player only sees what they're entitled to see:

private fun isZoneVisibleTo(zoneKey: ZoneKey, viewingPlayerId: EntityId): Boolean {
    return when (zoneKey.zoneType) {
        ZoneType.LIBRARY -> false  // Hidden from everyone
        ZoneType.HAND -> zoneKey.ownerId == viewingPlayerId  // Only owner sees
        ZoneType.BATTLEFIELD,
        ZoneType.GRAVEYARD,
        ZoneType.STACK,
        ZoneType.EXILE -> true  // Public zones
    }
}

Your opponent's hand stays hidden. Scry effects only reveal cards to the player who scryed. The engine can stay pure while the server handles the messy real-world concerns.

The server module also houses scenario tests—integration tests that simulate actual gameplay with player inputs and outputs. These tests verify that cards interact correctly in realistic situations: casting spells, responding to triggers, making decisions. When a new card is added, a scenario test ensures it behaves correctly in context, not just in isolation.

The Web Client

The web client follows what we call the "dumb terminal" pattern. It renders the game state it receives. It captures user intent through clicks. That's it.

submitAction: (action) => {
    ws?.send(createSubmitActionMessage(action))
    set({ selectedCardId: null, targetingState: null })
}

When you tap a creature, the client doesn't validate whether it's summoning sick or whether you control it—it just sends "player wants to tap this entity" to the server. The server checks legality using the engine, updates state, broadcasts the result.

This eliminates an entire class of bugs where client and server disagree about game rules. If the rules change, you only update the engine. The client stays blissfully ignorant, just visualizing state and forwarding actions.

Each layer stays focused. The engine doesn't know about networking. The server doesn't know about React. The client doesn't know the rules.

Why Build This?

Because I like challenges and I like playing Magic. The comprehensive rules are a 30-year accretion of edge cases that somehow still form a coherent system—exactly the kind of puzzle I can't resist.

Without AI assistance, this would have stayed on my "someday" list forever. Instead, it's live. The layer system handles Humility correctly. Triggered abilities fire in the right order. You can actually draft with friends.

The engine is open source at github.com/wingedsheep/argentum-engine. Play at magic.wingedsheep.com.

— Vincent

Introduction

Most backend codebases don't start broken. They start small, coherent, and easy to change. Then time passes.

Features ship. Teams grow. Deadlines happen. And one day you realize that changing the checkout flow requires touching seven packages, three shared utilities, and a database table that four other features depend on. You're not sure what will break, so you test everything manually and deploy on a Friday afternoon with your finger hovering over the rollback button.

This is two problems working together. Low cohesion: related code is scattered across the codebase instead of living together. High coupling: unrelated modules depend on each other's internals, so changes ripple outward in unpredictable ways.

When it gets bad enough, teams usually respond in one of two ways.

Some reach for microservices. Split everything into separate services. Coupling becomes impossible because nothing can import anything directly. But this trades code complexity for operational complexity: network failures become normal failures, data consistency requires careful choreography, and "running locally" becomes a research project. You wanted to fix your architecture and accidentally became a distributed systems team.

Others try better discipline. Add architectural guidelines. Improve folder structure. Be more careful in code review. This is cheaper and less disruptive—but discipline erodes. Someone imports an internal class because it's convenient. Someone adds a "temporary" dependency to meet a deadline. Guidelines without enforcement are aspirations.

A modular monolith is a way to make the second approach actually stick. It's a monolith where boundaries are enforced by the build system—where you can't import another module's internals because the compiler won't let you.

You keep the operational simplicity of a single deployable. But you get real architecture: explicit contracts between modules, isolated data, and the ability to change one module without accidentally breaking others. When something goes wrong at 3am, there's one service to check, one log to read, one thing to restart.

This guide is a practical walkthrough of building one in Kotlin and Spring Boot. I'll cover where to draw boundaries, how to enforce them with Gradle, how to structure code inside modules, how to isolate data, how modules communicate, error handling, and testing. The focus is on patterns that work in real codebases—things I've learned building systems this way, not theoretical ideals from a whiteboard.

Part 1: Where to Draw the Lines

We want boundaries. But where should they go?

Domain-Driven Design has opinions about this. DDD has a reputation for academic terminology and books thicker than your laptop, but ignore all that—the part that matters for architecture is simple: organize around business domains, not technical layers.

Domains, Not Layers

Most backend projects start with a structure like this:

com.example.app/
├── controllers/
│   ├── ProductController.kt
│   ├── OrderController.kt
│   └── ShippingController.kt
├── services/
│   ├── ProductService.kt
│   ├── OrderService.kt
│   └── ShippingService.kt
└── repositories/
    ├── ProductRepository.kt
    ├── OrderRepository.kt
    └── ShippingRepository.kt

All controllers in one folder, all services in another, all repositories in a third. It feels tidy. It looks professional in code review.

It's also a mistake.

When you add a shipping feature, you touch files in three different folders. When you want to understand how shipping works, you're jumping between layers, assembling the picture from scattered pieces. And nothing stops OrderService from calling ProductRepository directly—the folder structure creates the appearance of organization without any actual boundaries.

The alternative is organizing by domain:

com.example.app/
├── products/
│   ├── ProductController.kt
│   ├── ProductService.kt
│   └── ProductRepository.kt
├── orders/
│   ├── OrderController.kt
│   ├── OrderService.kt
│   └── OrderRepository.kt
└── shipping/
    ├── ShippingController.kt
    ├── ShippingService.kt
    └── ShippingRepository.kt

Now a shipping/ folder contains everything about shipping. New developers can open one folder and understand one capability. Changes stay local. And the structure reveals where real boundaries could be enforced.

The Same Word, Different Meanings

Ask three people in an e-commerce company what "Product" means:

The catalog team says name, description, price, images, sustainability rating—it's what customers browse and search. The warehouse team says weight, dimensions, and whether it's fragile—they need to put it in a box, not describe it. The inventory team says a product ID and a quantity on a shelf—they don't care what it looks like.

These aren't three views of the same thing. They're three different concepts that happen to share a name. DDD calls this a bounded context: a boundary within which a term has a consistent meaning. You can call it whatever you want. The point is that "Product" means something different depending on who you ask.

The instinct is to create one Product class that serves everyone:

data class Product(
    val id: ProductId,
    val name: String,
    val description: String,
    val price: Money,
    val images: List<Image>,
    val sustainabilityRating: String,
    val weightGrams: Int,
    val dimensions: Dimensions,
    val isFragile: Boolean,
    val warehouseQuantities: Map<WarehouseId, Int>,
    // ... and it keeps growing
)

One class, no duplication—efficient, right? But this is how coupling starts. The catalog team adds images; now warehouse code depends on it. The warehouse team adds dimensions; now catalog code carries that weight. Everyone's afraid to touch this class, and everyone has to.

The fix is recognizing that each context should define its own model:

// In the catalog context
data class Product(
    val id: ProductId,
    val name: String,
    val description: String,
    val price: Money,
    val images: List<Image>,
)

// In the shipping context
data class ShippableItem(
    val productId: ProductId,
    val weightGrams: Int,
    val dimensions: Dimensions,
    val isFragile: Boolean,
)

Yes, you now have two classes with some overlapping fields. This feels wasteful until you realize the alternative is a God object that grows until it collapses under its own weight. Duplication between contexts is healthy. Duplication within a context is a code smell. Learn the difference.

Finding the Right Boundaries

There's no algorithm for this, but there are useful signals.

Listen for different vocabulary. When warehouse staff say "pick" and "pack" while marketing says "browse" and "wishlist," you're hearing two contexts. Language differences usually reflect model differences—this is more reliable than most technical heuristics.

Different rates of change are another clue. Pricing rules might change weekly; shipping carrier integrations change quarterly. Bundling them means every pricing change risks breaking shipping. And if different people are responsible for different areas, those are natural seams. Conway's Law isn't just an observation—it's a force you can work with instead of against.

Transaction boundaries help too. If two operations almost always happen together in the same database transaction, they probably belong together. If they don't need transactional consistency, that's a hint they could be separate. Conversely, if module A makes fifteen calls to module B for every operation, maybe they shouldn't be separate at all.

Size matters. Can you describe the module in one sentence without conjunctions? "Manages the product catalog" works. "Handles payments and shipping and user profiles" is three modules wearing a trench coat.

Watch out for a few traps. Don't model from the database up—tables reflect storage decisions, not business boundaries. Don't share entities across contexts just because it feels DRY; if you're adding fields that only one context uses, you've got coupling. Don't slice too thin—you don't need a separate module for "OrderValidation," that's just part of orders. And don't draw boundaries that cut across team ownership. The best technical boundary won't help if nobody knows who's responsible for it.

The Example We'll Use

For this guide, we're building an e-commerce system with four bounded contexts:

Products owns the catalog—names, descriptions, prices, images. Orders handles purchases and their lifecycle. Shipping manages fulfillment and delivery. Inventory tracks what's in stock and where.

Orders depends on Products (needs to know what's being ordered) and Inventory (needs to check availability). Shipping depends on Orders (needs to know what to ship) and Products (needs weights and dimensions). Nobody depends on Orders—it's a leaf in the dependency graph.

These boundaries aren't perfect. We'll discover friction as we build. That's fine—the enforcement mechanisms in the next section make refactoring boundaries a manageable task rather than an archaeological expedition.

Part 2: Enforcing the Lines

Boundaries on a whiteboard are aspirations. Boundaries in the build system are architecture.

Without enforcement, boundaries erode. It happens slowly, always with good intentions. Someone imports an internal class because it's convenient. Someone adds a "temporary" dependency to meet a deadline. Six months later, your modules are coupled in ways nobody intended, and untangling them is a project of its own.

The solution is to make invalid dependencies a compiler error, not a code review discussion.

Contracts, Not Implementations

When Shipping needs product information, the obvious approach is to call the Products service directly:

class ShippingService(
    private val productService: ProductServiceImpl
) {
    fun calculateWeight(productId: ProductId): Grams {
        val product = productService.getProduct(productId)
        return product.weightGrams
    }
}

This works, but it creates tight coupling. Shipping now depends on ProductServiceImpl—a concrete class with its own dependencies, internal structure, and implementation details. And the coupling is transitive: ProductServiceImpl depends on ProductRepository, which depends on database entities. Shipping has indirectly coupled itself to the Products database schema.

The fix is depending on a contract instead of an implementation:

interface ProductServiceApi {
    fun getProduct(id: ProductId): ProductDto
}

class ProductServiceImpl(
    private val repository: ProductRepository
) : ProductServiceApi {
    override fun getProduct(id: ProductId): ProductDto { ... }
}

class ShippingService(
    private val productService: ProductServiceApi  // Interface, not implementation
) {
    fun calculateWeight(productId: ProductId): Grams {
        val product = productService.getProduct(productId)
        return Grams(product.weightGrams)
    }
}

Now Shipping depends on ProductServiceApi—an interface with no implementation details. The Products team can refactor their internals, change their database, swap out libraries. As long as they fulfill the contract, Shipping won't notice.

But where does the contract live? If the interface sits inside the Products module alongside its implementation, Shipping still depends on the Products module. We need to separate the contract into its own place.

The API/Implementation Split

In Gradle, we split each bounded context into two modules:

products/
├── products-api/     # The contract
└── products-impl/    # The implementation

The rules are simple. An -impl module depends on its own -api (it implements the contract). Other modules depend only on -api modules, never on -impl. No circular dependencies.

// shipping-impl/build.gradle.kts
dependencies {
    implementation(project(":shipping:shipping-api"))
    implementation(project(":products:products-api"))  // Contract only
    // Cannot add products-impl - that's the whole point
}

If someone tries to import a class from products-impl, the build fails. No discussion needed.

The -api module contains the public contract: interfaces defining what the module can do, DTOs for data exchange, events other modules might listen to, and error types so callers know what can go wrong.

// products-api
interface ProductServiceApi {
    fun getProduct(id: ProductId): Result<ProductDto, ProductError>
}

data class ProductDto(
    val id: ProductId,
    val name: String,
    val weightGrams: Int,
)

sealed class ProductError {
    data class NotFound(val id: ProductId) : ProductError()
}

The -impl module contains everything private: domain models with business logic, service implementations, persistence layer, controllers. Mark these internal so Kotlin reinforces the boundary:

// products-impl
@Service
internal class ProductServiceImpl(
    private val repository: ProductRepository,
) : ProductServiceApi {
    // Maps between internal domain model and public DTOs
}

internal data class Product(
    val id: ProductId,
    val name: String,
    val price: Money,
) {
    init {
        require(name.isNotBlank()) { "Name required" }
    }
}

Foreign References

There's a subtle trap when defining contracts. Consider an event that Shipping publishes when a package is delivered:

// shipping-api
data class ShipmentDeliveredEvent(
    val shipmentId: ShipmentId,
    val orderId: OrderId,  // Where does this come from?
)

OrderId lives in orders-api. So shipping-api depends on orders-api. Now if Orders wants to track shipment status and needs ShipmentId, you have a cycle between API modules. Gradle won't compile it.

The instinct is to extract both ID types to a common module. That works initially. Then Inventory needs ProductId. Then Notifications needs CustomerId. Soon your common module contains every ID type in the system, and you've recreated coupling with extra steps.

The fix: each module only defines types it owns. For foreign references, you could use primitives. Or, if you want to prevent passing a shipment id into an order id field by accident, you can create a reference type like OrderReference that is defined locally in the shipping module.

// shipping-api
@JvmInline value class OrderReference(val value: String)

data class ShipmentDeliveredEvent(
    val shipmentId: ShipmentId,
    val orderReference: OrderReference,
    val deliveredAt: Instant,
)

Type safety without coupling. The consuming code converts at the boundary:

// orders-impl
@Component
internal class ShippingEventListener(
    private val orderService: OrderService,
) {
    fun on(event: ShipmentDeliveredEvent) {
        val orderId = OrderId(event.orderReference.value)
        orderService.markAsDelivered(orderId, event.deliveredAt)
    }
}

This pattern makes sense for Shipping and Orders because they're peers—each might reference the other. But not every relationship is bidirectional. Invoicing is always for an order, but orders never reference invoices. The dependency only flows one way, so invoicing-api can depend on orders-api and use OrderId directly if you are more pragmatic about this.

Keeping Bounded Contexts Clean

With primitives at the boundary, you won't hit Gradle cycles. But there's a subtler concern: conceptual pollution.

Products is a foundational module—it defines what you can buy. Orders tracks purchases. Conceptually, Orders needs to know about Products (you order something), but Products shouldn't need to know about Orders. A product exists independently of whether anyone ordered it.

This isn't a technical constraint—with primitives, you could make Products depend on Orders. But you shouldn't. Every dependency accumulates knowledge. When Products depends on Orders, the Products team needs to understand order concepts to work on their module.

Think of modules as layers. Foundational modules at the bottom know nothing about modules above them:

When an arrow would point upward, pause and reconsider. Usually there's an event that could flow the other direction, or an SPI waiting to be extracted.

Service Provider Interfaces

Sometimes a foundational module needs input from higher-level modules without depending on them. Products wants to prevent deletion of a product with pending orders—but shouldn't know about Orders.

The solution is a Service Provider Interface (SPI). Products defines an interface for the question it wants to ask. Other modules provide answers by implementing it.

// products-api/spi/ProductDeletionBlocker.kt
interface ProductDeletionBlocker {
    fun canDelete(productId: String): Boolean
}

Orders implements it:

// orders-impl
@Service
internal class OrderBasedDeletionBlocker(
    private val orderRepository: OrderRepository
) : ProductDeletionBlocker {
    override fun canDelete(productId: String): Boolean {
        return !orderRepository.existsPendingForProduct(ProductId(productId))
    }
}

Products consumes all implementations without knowing where they come from:

// products-impl
@Service
internal class ProductServiceImpl(
    private val repository: ProductRepository,
    private val deletionBlockers: List<ProductDeletionBlocker>
) : ProductServiceApi {
    
    fun deleteProduct(id: ProductId): Result<Unit, ProductError> {
        if (deletionBlockers.any { !it.canDelete(id.value) }) {
            return Err(ProductError.DeletionBlocked)
        }
        repository.delete(id)
        return Ok(Unit)
    }
}

Spring collects all beans implementing ProductDeletionBlocker and injects them as a list. Products doesn't know who's blocking or why—it just asks. The knowledge stays where it belongs.

Source Sets Count Too

Gradle projects have test and testFixtures source sets, each with their own dependencies. Each can quietly violate your architecture.

The trap: you create buildProduct() in products-impl:testFixtures. Orders tests need products too, and there's a perfectly good builder right there. So orders-impl:testFixtures depends on products-impl:testFixtures. Now your Orders tests are coupled to Products' internal implementation.

Test fixtures follow the same rules as main code. If a-impl:main can't depend on b-impl:main, then a-impl:testFixtures can't depend on b-impl:testFixtures. The duplication this creates is intentional—each module's test fixtures stay self-contained.

Automated Enforcement

Gradle modules prevent most violations—you can't import what you can't depend on. But some rules need explicit checks, like ensuring no -api module depends on an -impl module, or that test fixtures respect boundaries.

Write a validation task that fails the build on violations, and run it as part of CI. Check the Gradle plugin from the example code: [link]

Architecture enforced by the build system survives deadlines, new team members, and "temporary" workarounds. Architecture enforced by documentation survives until the first Thursday afternoon crunch.

Alternative: Spring Modulith

The Gradle multi-module approach provides the strongest guarantees, but Spring Modulith offers a lighter-weight alternative using package structure and test-time verification.

Spring Modulith treats each top-level package as a module. Classes directly in a module's package are its API; anything in subpackages is internal:

com.example.app/
├── products/
│   ├── ProductService.kt        # API - accessible
│   └── internal/                # Internal - hidden
│       └── ProductRepository.kt
└── shipping/
    └── ...

A test verifies the structure:

@Test
fun `verify module structure`() {
    ApplicationModules.of(Application::class.java).verify()
}

Spring Modulith also provides automatic documentation generation, @ApplicationModuleTest for isolated module testing, and an event publication registry that handles the transactional outbox pattern.

The tradeoff is enforcement timing. Gradle modules reject invalid imports at compile time. Spring Modulith catches them at test time. Stronger guarantees versus easier setup.

You can combine them—use Gradle modules for hard boundaries, add Spring Modulith for its testing and documentation features. Start with your situation: greenfield with clear boundaries favors Gradle modules; existing monolith with unclear boundaries favors Spring Modulith to discover structure first.

Part 3: Inside the Modules

We have modules with enforced boundaries. The build system prevents coupling. The hard part is done.

What happens inside each -impl module matters less now. A mess in one module can't leak into others. You can refactor later without coordinating across teams. Internal structure is a local decision.

That said, one principle is worth following: dependencies point inward.

The Three Layers

Organize code so that outer layers depend on inner layers, never the reverse.

Domain is the core: business logic, entities, value objects, repository interfaces. No framework dependencies—just plain Kotlin. The domain defines what the system does, not how it connects to the outside world.

Application orchestrates: it implements the API contract, coordinates domain operations, handles transactions, publishes events. Application code uses domain types and repository interfaces but doesn't know about databases or HTTP.

Infrastructure connects to the outside world: controllers, repository implementations, message consumers, external API clients. This is where Spring annotations live, where SQL gets written, where HTTP requests get parsed.

The dependency direction: infrastructure → application → domain. A request flows inward: the controller (infrastructure) calls a handler (application), which uses domain types and a repository interface. The repository implementation (infrastructure) knows how to persist those types—but the domain doesn't know the implementation exists.

This keeps your business logic testable without frameworks and portable across different infrastructure choices. Need to swap Postgres for MongoDB? Only the repository implementation changes. The domain doesn't care.

What Stays Internal

The -api module contains only what other modules actually use. If no other module needs ProductId, it stays in -impl. Don't preemptively publish types "just in case."

That said, value types, enums, and DTOs with simple init validation often do belong in -api. Other modules benefit from knowing they're working with a valid ProductId, not just a String.

// products-api
data class ProductId(val value: String) {
    init {
        require(value.isNotBlank()) { "ProductId cannot be blank" }
    }
}

enum class ProductStatus { DRAFT, ACTIVE, DISCONTINUED }

data class ProductDto(
    val id: ProductId,
    val name: String,
    val status: ProductStatus,
    val priceInCents: Long,
)

What stays internal is behavior. Business rules, state transitions, aggregate invariants—the logic that makes your domain more than a data container:

// products-impl
internal data class Product(
    val id: ProductId,
    val name: String,
    val price: Money,
    val status: ProductStatus,
) {
    fun applyDiscount(percent: Int): Product = 
        copy(price = price.discountBy(percent))
    
    fun discontinue(): Product {
        check(status != ProductStatus.DISCONTINUED) { "Already discontinued" }
        return copy(status = ProductStatus.DISCONTINUED)
    }
}

internal fun Product.toDto() = ProductDto(
    id = id,
    name = name,
    status = status,
    priceInCents = price.toCents(),
)

The rule of thumb: if it's just data and validation, it can be public. If it's behavior, keep it internal.

Organizing Inside the Module

This is less important than it seems. The module boundary protects you—a mess inside one module can't leak into others. Refactor your internal structure whenever you want without coordinating with anyone.

That said, when a module grows multiple features, vertical slices work well. Each feature gets a subpackage with its domain, persistence, messaging, and REST concerns. Shared concepts stay in the module root.

products-impl/
└── src/main/kotlin/com/example/products/
    ├── Product.kt                    # Shared domain
    ├── ProductRepository.kt          # Shared persistence
    ├── ProductServiceFacade.kt       # API implementation
    ├── import/
    │   ├── ImportHandler.kt
    │   ├── ImportJob.kt
    │   ├── persistence/
    │   ├── messaging/
    │   └── rest/
    └── pricing/
        ├── PricingHandler.kt
        ├── PriceCalculator.kt
        ├── persistence/
        ├── messaging/
        └── rest/

When you need to understand pricing, you open one folder. But don't overthink this—pick something reasonable and move on. You can always reorganize later.

Connecting to the API Contract

Other modules depend on the -api interface. A facade delegates to handlers:

@Service
internal class ProductServiceFacade(
    private val importHandler: ImportHandler,
    private val pricingHandler: PricingHandler,
) : ProductServiceApi {

    override fun importProducts(request: ImportRequest) = 
        importHandler.handle(request)
    
    override fun updatePrice(id: ProductId, request: UpdatePriceRequest) = 
        pricingHandler.handle(id, request)
}

Pure delegation, no logic. Other modules see one interface; internally, work is split by feature. Controllers inject handlers directly since they're in the same module.

When to Skip All This

Hexagonal architecture and use-case slicing add value when you have complex domain logic that benefits from isolation, multiple entry points (REST, messaging, CLI) to the same operations, or a need to swap infrastructure components.

They add overhead when you're building straightforward CRUD, when the module is small and unlikely to grow, or when you're prototyping and don't know what the domain looks like yet.

Start simple. Add structure when the code tells you it needs it—when you're afraid to touch a class because it does too many things, or when testing requires mocking half the framework. Structure is a response to pain, not a prerequisite.

Part 4: Isolating Data

You can have perfectly separated modules, clean APIs, and enforced dependencies—and still end up with a tightly coupled system. The culprit? The database.

When modules share tables, they share problems. When one module writes directly to another's tables, your boundaries exist only in your imagination. When a foreign key reaches across module boundaries, you've created a dependency that no Gradle configuration can catch.

The Shared Database Trap

It usually starts innocently. The Shipping module needs product weights. Products already has a products table. Why not just join?

-- In shipping code
SELECT s.*, p.weight_grams
FROM shipments s
JOIN products p ON s.product_id = p.id
WHERE s.id = ?

This works. It's fast. It's "just one query."

It's also invisible coupling. Now Shipping depends on the Products table structure. If Products renames weight_grams, Shipping breaks. If Products moves to a different database, Shipping breaks. And nobody sees this dependency in the code—it lurks in SQL strings, waiting to cause an incident during an otherwise routine deployment.

Schema Per Module

The fix is giving each module its own database schema. Products owns the products schema. Shipping owns the shipping schema. The module can only touch tables in its own schema.

CREATE SCHEMA products;
CREATE SCHEMA shipping;

CREATE TABLE products.product (
    id VARCHAR(255) PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    weight_grams INT NOT NULL
);

CREATE TABLE shipping.shipment (
    id VARCHAR(255) PRIMARY KEY,
    product_id VARCHAR(255) NOT NULL,  -- Just data, no FK
    weight_grams INT NOT NULL           -- Copied at creation time
);

Each module gets its own Flyway configuration pointing at its schema. Migrations live with the module code. When there's a problem with the shipments table, there's no ambiguity about who owns it. You can check the example implementation here.

This isn't just convention—configure your repositories to only access their schema, and cross-schema queries become impossible rather than discouraged.

No Foreign Keys Across Schemas

This is the rule that makes people uncomfortable: no foreign keys between schemas.

-- Don't do this
CREATE TABLE shipping.shipment (
    id VARCHAR(255) PRIMARY KEY,
    product_id VARCHAR(255) REFERENCES products.product(id)  -- No!
);

-- Do this instead
CREATE TABLE shipping.shipment (
    id VARCHAR(255) PRIMARY KEY,
    product_id VARCHAR(255) NOT NULL  -- Just a string
);

Referential integrity across module boundaries becomes your responsibility at the application level. When Shipping creates a shipment, it validates that the product exists by calling the Products API:

override fun createShipment(request: CreateShipmentRequest): Result<ShipmentDto, ShipmentError> {
    val product = productService.getProduct(request.productId)
        .getOrElse { return Err(ShipmentError.ProductNotFound(request.productId)) }

    val shipment = Shipment(
        id = ShipmentId.generate(),
        productId = request.productId,
        weightGrams = product.weightGrams,
    )

    return Ok(shipmentRepository.save(shipment).toDto())
}

Yes, this is more work than a foreign key. But it's explicit—visible in the code, testable, and under your control. And when you eventually need to extract a module into its own service, there are no cross-schema constraints to untangle.

What About Joins?

The most common concern: "I used to join Orders and Products in one query. Now what?"

For single-item lookups, call both services and combine the results. For lists, batch your calls—collect all product IDs first, fetch them in one bulk call, then join in memory:

fun getOrderSummaries(orderIds: List<OrderId>): List<OrderSummary> {
    val orders = orderService.getOrders(orderIds)
    val productIds = orders.map { it.productId }.distinct()
    val products = productService.getProducts(productIds).associateBy { it.id }
    
    return orders.map { order ->
        OrderSummary(order, products[order.productId])
    }
}

For historical accuracy, denormalize at write time. An order should show the product name and price at the time of purchase, not whatever the product is called today. Copy the relevant fields when creating the order. The data belongs to the order now—it's capturing a moment in time.

For analytics and reporting, pragmatism wins. A read-only reporting schema with cross-module views is fine for dashboards—just keep it separate from your application code.

The Tradeoff

Data isolation requires more code. You lose database-enforced referential integrity across modules. If a product is deleted while a shipment references it, the database won't stop you—your application code has to handle that.

What you get is independence. Products can restructure its tables without Shipping noticing. Module tests can use a real database for their own schema while mocking other modules' APIs. And if you ever extract a module into its own service, the path is clear—no foreign keys to remove, no shared tables to split.

Part 5: Module Communication

Modules need to talk to each other. An order needs product information. A shipment needs to know when payment completes. The question isn't whether modules communicate—it's how they communicate without reintroducing the coupling we worked so hard to eliminate.

Two patterns cover most cases: synchronous calls when you need an answer now, and asynchronous events when you're announcing something happened.

Synchronous Calls

The simplest pattern: one module calls another's API and waits for the response.

@Service
internal class ShipmentServiceImpl(
    private val productService: ProductServiceApi,
    private val shipmentRepository: ShipmentRepository,
) : ShipmentServiceApi {

    override fun createShipment(request: CreateShipmentRequest): Result<ShipmentDto, ShipmentError> {
        val product = productService.getProduct(request.productId)
            .getOrElse { return Err(ShipmentError.ProductNotFound(request.productId)) }

        val shipment = Shipment(
            id = ShipmentId.generate(),
            productId = request.productId,
            weightGrams = product.weightGrams,
            status = ShipmentStatus.PENDING,
        )

        return Ok(shipmentRepository.save(shipment).toDto())
    }
}

This is appropriate when you need data to proceed and the operation should fail if the dependency fails. The tradeoff is runtime coupling—if Products is slow, Shipping is slow. For many operations, that's exactly right.

When the external module returns data shaped for their needs rather than yours, translate it at the boundary. Shipping doesn't need product descriptions or images—it needs weight and dimensions. Create an adapter that fetches what you need and discards the rest:

@Component
internal class ProductAdapter(
    private val productService: ProductServiceApi
) {
    fun getShippableItem(productId: ProductId): ShippableItem? {
        val dto = productService.getProduct(productId).getOrNull() ?: return null
        
        return ShippableItem(
            id = productId,
            weight = Weight.fromGrams(dto.weightGrams),
            dimensions = Dimensions(dto.width, dto.height, dto.depth),
            isFragile = dto.tags.contains("FRAGILE"),
        )
    }
}

Now your domain code works with ShippableItem—a type you control—instead of ProductDto which might change when the Products team refactors.

Asynchronous Events

Sometimes a module doesn't need a response. It's announcing that something happened, and other modules react if they care.

// payment-api
data class PaymentCompletedEvent(
    val paymentId: String,
    val orderReference: String,
    val amountInCents: Long,
    val currency: String,
    val timestamp: Instant,
)

The publisher doesn't know who's listening. Shipping might start fulfillment. Inventory might confirm a reservation. Notifications might send an email. Each listener is independent.

For in-process events, Spring's ApplicationEventPublisher works fine:

@Service
internal class PaymentServiceImpl(
    private val eventPublisher: ApplicationEventPublisher,
) : PaymentServiceApi {

    @Transactional
    override fun completePayment(paymentId: PaymentId): Result<PaymentDto, PaymentError> {
        val payment = // ... complete the payment
        
        eventPublisher.publishEvent(PaymentCompletedEvent(
            paymentId = payment.id.value,
            orderId = payment.orderId.value,
            amountInCents = payment.amount.toCents(),
            currency = payment.amount.currency.code,
            timestamp = Instant.now(),
        ))
        
        return Ok(payment.toDto())
    }
}

Listeners react in their own modules:

// shipping-impl
@Component
internal class PaymentEventListener(
    private val shipmentService: ShipmentService,
) {
    @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
    fun on(event: PaymentCompletedEvent) {
        val orderId = OrderId(event.orderId)
        shipmentService.startFulfillment(orderId)
    }
}

The AFTER_COMMIT phase ensures the listener only fires if the payment transaction actually commits. No point starting fulfillment for a payment that rolled back.

The Dual-Write Problem

There's a subtle issue with publishing events directly. Consider this sequence:

1. Update database (payment marked complete)
2. Publish event (notify listeners)
3. Return success

What if the database write succeeds but event publishing fails? The payment is complete, but nobody knows. Shipping never starts.

What if the event publishes but then the transaction rolls back? Listeners react to something that didn't actually happen.

This is the dual-write problem: updating two systems without a shared transaction.

The solution is making event publishing part of the database transaction, using a transactional outbox. Instead of publishing directly, write the event to an outbox table in the same transaction as your business data.

@Transactional
override fun completePayment(paymentId: PaymentId): Result<PaymentDto, PaymentError> {
    val payment = // ... complete the payment
    
    outboxRepository.save(OutboxEvent(
        id = EventId.generate(),
        type = "PaymentCompleted",
        payload = Json.encodeToString(PaymentCompletedEvent(...)),
        createdAt = Instant.now(),
    ))
    
    return Ok(payment.toDto())
}

A separate process polls the outbox and publishes to listeners. If the business transaction rolls back, the outbox entry rolls back too. If it commits, the event is guaranteed to be published eventually.

This guarantees at-least-once delivery. If the publisher crashes after sending but before marking the entry as processed, it republishes on restart. Listeners need to handle duplicates—make them idempotent.

Spring Modulith provides this out of the box with its event publication registry. Add spring-modulith-starter-jpa and events are automatically persisted and retried on failure. Worth considering before building your own.

When to Use What

Most module communication is synchronous calls—simple and direct. Use them when you need data to proceed.

Events handle reactions and loose coupling. Use them when multiple modules might care about something happening, or when the caller shouldn't wait for downstream processing.

The outbox adds reliability when you can't afford to lose events. For in-process events where "probably delivered" is good enough, direct publishing is fine.

The patterns aren't mutually exclusive. A checkout flow might make synchronous calls to validate inventory, write the order and an outbox entry in one transaction, then have listeners react asynchronously to start fulfillment.

Part 6: Errors and Validation

Errors are where modular codebases quietly rot. One team throws exceptions, another returns nulls, a third invents custom result types. The controller layer becomes a graveyard of catch blocks trying to translate chaos into HTTP status codes.

Make Errors Explicit

The traditional approach:

fun getProduct(id: ProductId): Product {
    return repository.findById(id)
        ?: throw ProductNotFoundException(id)
}

Nothing in the type signature hints that this might throw. You discover ProductNotFoundException exists when it crashes in production—or if you're lucky, by reading documentation that probably doesn't exist.

But "product not found" isn't exceptional. It's a normal business case—the user typed a wrong ID, the product was deleted. This happens all the time, and the code should make it obvious.

Return errors as values instead. Define them in the -api module using sealed classes:

// products-api
sealed class ProductError {
    data class NotFound(val id: ProductId) : ProductError()
    data class InvalidData(val reason: String) : ProductError()
    data object DuplicateName : ProductError()
}

interface ProductServiceApi {
    fun getProduct(id: ProductId): Result<ProductDto, ProductError>
    fun createProduct(request: CreateProductRequest): Result<ProductDto, ProductError>
}

Now the type signature tells you everything. And because ProductError is sealed, the compiler knows all possible cases. Add a new error type and every call site that doesn't handle it becomes a compile error.

// products-impl
@Service
internal class ProductServiceImpl(
    private val repository: ProductRepository,
) : ProductServiceApi {

    override fun getProduct(id: ProductId): Result<ProductDto, ProductError> {
        val product = repository.findById(id)
            ?: return Err(ProductError.NotFound(id))
        return Ok(product.toDto())
    }
}

Callers handle both cases explicitly:

productService.getProduct(productId).fold(
    onSuccess = { product -> /* use it */ },
    onFailure = { error -> /* handle it */ }
)

No try-catch. No wondering what might throw.

A note on Result types: Kotlin's built-in Result only supports Throwable as the error type—it can't model domain errors. You need a proper either type with Ok and Err variants. The example project includes a simple implementation, or use kotlin-result or Arrow's Either. Pick one and use it everywhere.

Validation in Init Blocks

Where should validation live? The traditional answer in Spring is JSR-303 annotations—@NotBlank, @Positive, @Valid. It works, but scatters validation rules across annotations that are easy to miss and harder to test than plain code.

A better approach: validate at construction time. If an object exists, it's valid. This follows the "Parse, don't validate" principle—your init blocks act as a firewall, preventing garbage data from ever entering your implementation layer.

Kotlin gives you require for validating inputs and check for validating state. Both throw on failure, but they signal different problems. require throws IllegalArgumentException—the caller passed bad input. check throws IllegalStateException—something is wrong internally.

Structural vs. Business Validation

Not all validation belongs in the same place:

Structural validation in API modules works well: exceptions fire immediately during deserialization, the code documents requirements for consumers, and your domain logic doesn't need defensive null checks everywhere.

// orders-api
data class CreateOrderRequest(
    val customerId: String,
    val items: List<OrderItemDto>,
) {
    init {
        require(customerId.isNotBlank()) { "Customer ID is required" }
        require(items.isNotEmpty()) { "Order must contain at least one item" }
    }
}

data class OrderItemDto(
    val productId: String,
    val quantity: Int
) {
    init {
        require(productId.isNotBlank()) { "Product ID is required" }
        require(quantity > 0) { "Quantity must be positive" }
    }
}

Avoid encoding volatile rules in the API. require(password.isNotEmpty()) is fine; require(password.length >= 10) couples your contract to a policy that might change. Check specific length requirements in the service layer.

For internal domain objects, use check:

// products-impl
internal data class Product(
    val id: ProductId,
    val name: String,
    val price: Money,
) {
    init {
        check(name.isNotBlank()) { "Product name cannot be blank" }
        check(name.length <= 200) { "Product name too long" }
    }
}

If these fail, it means your application code tried to create an invalid domain object—a bug, not bad user input.

Translating Errors to HTTP

The domain layer doesn't know about HTTP. It returns ProductError.NotFound. Somewhere, that needs to become a 404.

That translation happens at the controller:

@RestController
internal class ProductController(
    private val productService: ProductServiceApi,
) {
    @GetMapping("/products/{id}")
    fun getProduct(@PathVariable id: ProductId): ResponseEntity<ProductDto> {
        return productService.getProduct(id).fold(
            onSuccess = { ResponseEntity.ok(it) },
            onFailure = { throw it.toResponseStatusException() }
        )
    }
}

fun ProductError.toResponseStatusException() = when (this) {
    is ProductError.NotFound -> ResponseStatusException(NOT_FOUND, "Product not found: $id")
    is ProductError.InvalidData -> ResponseStatusException(BAD_REQUEST, reason)
    is ProductError.DuplicateName -> ResponseStatusException(CONFLICT, "Product name already exists")
}

Spring Boot converts ResponseStatusException to RFC 7807 Problem Details automatically. The domain stays clean. The HTTP translation is explicit and in one place. When you add a new error type, the when expression forces you to decide what HTTP status it maps to.

A global exception handler deals with the init block validations:

@RestControllerAdvice
class GlobalExceptionHandler {

    @ExceptionHandler(IllegalArgumentException::class)
    fun handleBadInput(ex: IllegalArgumentException): ProblemDetail {
        return ProblemDetail.forStatusAndDetail(
            HttpStatus.BAD_REQUEST,
            ex.message ?: "Invalid request"
        )
    }

    @ExceptionHandler(IllegalStateException::class)
    fun handleInternalError(ex: IllegalStateException): ProblemDetail {
        // Log it - this is a bug
        return ProblemDetail.forStatusAndDetail(
            HttpStatus.INTERNAL_SERVER_ERROR,
            "Something went wrong"
        )
    }
}

When Exceptions Are Still Fine

This doesn't mean exceptions are always wrong. Use them for genuinely exceptional situations: database connection lost, file system full, external service unreachable. These aren't business cases your callers should handle individually—they're infrastructure failures that bubble up to a global handler.

The distinction: if the caller can reasonably do something about it, return a Result. If it's an unexpected failure that should abort the operation, throw

Part 7: Testing

A modular architecture should make testing easier, not harder. If you need to spin up the entire application to test whether a discount calculation works, something has gone wrong.

The module boundaries we've enforced create natural test boundaries. Each module has a clear API and explicit dependencies. This suggests a strategy: test one module completely, mock the others.

Module Tests

The Shipping module depends on ProductServiceApi, not ProductServiceImpl. In tests, provide a mock and test Shipping in isolation—real database, real transactions, real queries—without Products existing at all.

Gherkin works well for expressing what a module should do:

Feature: Shipment Creation

  Scenario: Create shipment for valid product
    Given a product "PROD-123" with weight 500 grams
    When I create a shipment for product "PROD-123" to "123 Main St"
    Then the shipment should be created with weight 500 grams
    And the shipment status should be "PENDING"

  Scenario: Fail when product does not exist
    Given no product "UNKNOWN" exists
    When I create a shipment for product "UNKNOWN" to "123 Main St"
    Then the shipment should fail with "ProductNotFound"

The step definitions mock external modules. "Given a product exists" doesn't create a real product—it stubs the mock to return one:

class ShipmentSteps(
    private val shippingService: ShippingServiceApi,
    private val productService: ProductServiceApi,  // Mocked
) {
    private var result: Result<ShipmentDto, ShipmentError>? = null

    @Given("a product {string} with weight {int} grams")
    fun givenProduct(productId: String, weight: Int) {
        whenever(productService.getProduct(ProductId(productId)))
            .thenReturn(Ok(buildProductDto(id = ProductId(productId), weightGrams = weight)))
    }

    @When("I create a shipment for product {string} to {string}")
    fun createShipment(productId: String, address: String) {
        result = shippingService.createShipment(
            CreateShipmentRequest(productId = ProductId(productId), address = address)
        )
    }

    @Then("the shipment should be created with weight {int} grams")
    fun verifyWeight(expectedWeight: Int) {
        assertThat(result?.value?.weightGrams).isEqualTo(expectedWeight)
    }
}

When a module test fails, you know which module broke. When it passes, you have confidence the module actually fulfills its contract against a real database. These are often your highest-value tests—fast enough to run frequently, realistic enough to catch real bugs.

Unit and Repository Tests

Module tests cover most scenarios, but some things deserve focused tests.

Complex logic that doesn't need a database—calculations, state machines, validation rules—these are faster to test in isolation:

class MoneyTest {
    @Test
    fun `discount reduces amount correctly`() {
        val money = Money(BigDecimal("100"), EUR)
        
        assertThat(money.discountBy(20).amount).isEqualByComparingTo(BigDecimal("80"))
    }
}

Complex queries where the logic lives in SQL also deserve their own tests. Filters, pagination, edge cases with NULL values—these are hard to get right and easy to break. Use Testcontainers to test against a real Postgres. Don't use H2 "because it's faster"—it behaves differently in ways that will bite you.

@DataJdbcTest
@Import(FlywayConfig::class)
class ProductRepositoryTest {

    companion object {
        @Container
        @ServiceConnection
        val postgres = PostgreSQLContainer("postgres:15")
    }
    
    @Autowired
    private lateinit var repository: ProductRepository
    
    @Test
    fun `finds products by category with price range`() {
        // Given
        repository.save(buildProduct(category = "electronics", priceInCents = 5000))
        repository.save(buildProduct(category = "electronics", priceInCents = 15000))
        repository.save(buildProduct(category = "clothing", priceInCents = 3000))
        
        // When
        val results = repository.findByCategoryAndPriceRange(
            category = "electronics", minPrice = 1000, maxPrice = 10000
        )
        
        // Then
        assertThat(results).hasSize(1)
        assertThat(results.first().priceInCents).isEqualTo(5000)
    }
}

If a method just does findById or delegates to a repository and maps the result, skip the dedicated test—the module test already covers it.

Application-Level Tests

Sometimes you need to verify the full flow—all modules wired together, real database, real event publishing. Save these for critical user journeys where integration failures would be costly.

Feature: Checkout

  Scenario: Complete purchase creates order and schedules shipment
    Given I am logged in as a customer
    And a product "Widget" priced at €29.99
    And the product is in stock
    When I place an order for 1 "Widget"
    Then an order should be created
    And a shipment should be scheduled with status "PENDING"

The step definitions wire this to real services—no mocks. These tests are slow and brittle. When they fail, the cause isn't always obvious. Write few of them and let module tests handle the detailed behavior.

Test Fixtures

Tests need data. You don't want every test constructing objects from scratch, but you also don't want shared fixtures that create invisible dependencies between tests.

Builder functions with default arguments hit the sweet spot:

fun buildProductDto(
    id: ProductId = ProductId(UUID.randomUUID().toString()),
    name: String = "Test Product",
    weightGrams: Int = 100,
) = ProductDto(id = id, name = name, weightGrams = weightGrams)

// Only specify what matters for this test
val lightweight = buildProductDto(weightGrams = 50)

When you read a test that says buildProductDto(weightGrams = 500), you know the weight matters. Everything else is scaffolding.

Keep fixtures independent across modules. Shipping tests mock ProductServiceApi and use buildProductDto() from the -api test fixtures. They don't reach into Products' internal test helpers. The same boundary rules apply to test code.

Test Behavior, Not Implementation

Ask "what should happen" rather than "how does it happen internally."

// Good - tests outcome
@Test
fun `order fails when product is out of stock`() {
    whenever(inventoryService.checkAvailability(productId, quantity = 5))
        .thenReturn(Available(inStock = 3))
    
    val result = orderService.createOrder(CreateOrderRequest(productId, quantity = 5))
    
    assertThat(result.error).isEqualTo(OrderError.InsufficientStock(available = 3))
}

// Brittle - tests mechanics
@Test
fun `createOrder calls inventory then repository then publisher`() {
    orderService.createOrder(request)
    
    inOrder(inventoryService, orderRepository, eventPublisher).apply {
        verify(inventoryService).checkAvailability(any(), any())
        verify(orderRepository).save(any())
        verify(eventPublisher).publishEvent(any())
    }
}

The first test survives refactoring. Reorder the internal calls, add caching, change how you publish events—the test still passes because the behavior is the same. The second test breaks the moment you touch the implementation, even if nothing is actually wrong.

Don't chase coverage numbers. A codebase with 70% coverage and thoughtful tests beats one with 95% coverage and useless tests. Coverage tells you what code ran, not whether your tests would catch a bug.

Part 8: Putting It All Together

We've covered modules, boundaries, data isolation, communication, errors, and testing. How does it actually become a running application?

The good news: it's simpler than you might expect. One application module pulls in the implementations, Spring wires them together, and you deploy a single JAR.

The Application Module

The application module is the composition root—the place where all the pieces come together. Its build.gradle.kts depends on all the -impl modules:

// application/build.gradle.kts
dependencies {
    implementation(project(":products:products-impl"))
    implementation(project(":orders:orders-impl"))
    implementation(project(":shipping:shipping-impl"))
    implementation(project(":inventory:inventory-impl"))
    
    // Spring Boot, database drivers, etc.
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("org.postgresql:postgresql")
}

This is the only place that depends on -impl modules. Everyone else depends on -api modules. The application module breaks that rule because its job is to assemble everything.

The application class itself is just a standard Spring Boot entry point:

@SpringBootApplication(scanBasePackages = ["com.example"])
class Application

fun main(args: Array<String>) {
    runApplication<Application>(*args)
}

The scanBasePackages needs to cover your root namespace so Spring discovers components in all your -impl modules.

How Spring Wires It

A common question: if ShippingServiceImpl needs ProductServiceApi, and they're in different Gradle modules, how does Spring connect them?

Spring doesn't care about Gradle modules—it cares about the classpath. When the application starts, all the -impl modules are on the classpath. Spring scans for components, finds ProductServiceImpl (which implements ProductServiceApi), and registers it as a bean. When it creates ShippingServiceImpl, it sees a constructor parameter of type ProductServiceApi, finds the matching bean, and injects it.

// products-impl
@Service
internal class ProductServiceImpl(...) : ProductServiceApi

// shipping-impl
@Service
internal class ShippingServiceImpl(
    private val productService: ProductServiceApi  // Spring injects ProductServiceImpl
) : ShippingServiceApi

The interface is public (in -api). The implementation is internal (in -impl). Spring wires them together because at runtime, they're all in the same application context.

This is one of the key benefits of a modular monolith over microservices—no service discovery, no HTTP clients, no serialization overhead. Just dependency injection.

Project Structure

The overall structure follows naturally from what we've covered:

project-root/
├── build-logic/                    # Shared Gradle conventions
│   └── src/main/kotlin/
│       └── kotlin-conventions.gradle.kts
├── common/
│   ├── common-types/               # Shared value types
│   └── common-result/              # Result type
├── products/
│   ├── products-api/
│   └── products-impl/
├── orders/
│   ├── orders-api/
│   └── orders-impl/
├── shipping/
│   ├── shipping-api/
│   └── shipping-impl/
├── application/                    # Composition root
└── settings.gradle.kts

The build-logic module contains convention plugins that centralize shared Gradle configuration. Instead of repeating Kotlin version, test configuration, and dependencies in every build.gradle.kts, modules apply a plugin like id("kotlin-conventions"). Change the convention once, every module picks it up.

Use gradle/libs.versions.toml to manage dependency versions in one place:

[versions]
kotlin = "2.0.21"
spring-boot = "3.4.1"
testcontainers = "1.20.4"

[libraries]
spring-boot-starter-web = { module = "org.springframework.boot:spring-boot-starter-web", version.ref = "spring-boot" }
spring-boot-starter-data-jdbc = { module = "org.springframework.boot:spring-boot-starter-data-jdbc", version.ref = "spring-boot" }
testcontainers-postgres = { module = "org.testcontainers:postgresql", version.ref = "testcontainers" }

[plugins]
kotlin-jvm = { id = "org.jetbrains.kotlin.jvm", version.ref = "kotlin" }
spring-boot = { id = "org.springframework.boot", version.ref = "spring-boot" }

Then in any module:

dependencies {
    implementation(libs.spring.boot.starter.web)
    testImplementation(libs.testcontainers.postgres)
}

No more version strings scattered across dozens of build.gradle.kts files. Update a version once in the TOML, and every module picks it up.

Running It

The Spring Boot plugin packages everything into a single executable JAR. Run ./gradlew :application:bootJar and you get one artifact containing all modules, all dependencies, and an embedded server. Deploy it anywhere that runs Java.

For local development, you typically need a database. A simple Docker Compose handles that:

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
    ports:
      - "5432:5432"

Start the container, run ./gradlew :application:bootRun, and you're developing.

When You Outgrow It

The architecture is designed to be "split-ready," even if you never split.

If a module eventually needs to become a separate service, the path is clear. The database is already isolated—each module has its own schema, so you export it to a new database without untangling shared tables. The API is already defined—the -api module becomes the contract for the new service, and you replace the in-process implementation with an HTTP client. Communication is already explicit—you swap dependency injection for network calls. Tests are already structured—module tests that mocked ProductServiceApi now mock the HTTP client instead.

You're not refactoring a tangled mess. You're promoting a module that's already isolated.

Most teams never need to do this. The modular monolith scales further than people expect—both technically and organizationally. Multiple teams can work on different modules without stepping on each other, and the single deployment model remains manageable well into hundreds of thousands of lines of code.

But knowing the escape hatch exists makes the choice less risky. You're not betting everything on a monolith forever. You're choosing the simplest architecture that works today while keeping your options open.

Conclusion

A modular monolith isn't a compromise or a stepping stone to microservices. For most teams, it's the destination.

You get boundaries that hold—not because everyone remembered the guidelines, but because the compiler enforces them. You get changes that stay local, because modules can't reach into each other's internals. You get deployment that stays simple, because it's still one JAR, one process, one thing to monitor at 3am when something goes wrong.

Not everything in this guide carries equal weight. Some things are load-bearing: enforcement through Gradle modules, the API/implementation split, data isolation. Skip these, and the "modular" part of your monolith will erode within months. Other things—how you organize packages inside a module, whether you use Cucumber, what you name your handlers—are local decisions. Get them wrong and you have a mess, but it's a contained mess. The module boundary limits the blast radius.

That's the real payoff. In a traditional monolith, every shortcut becomes everyone's problem. In a modular monolith, a messy module is just a messy module. Clean it up later, on your own schedule, without coordinating across teams.

Build the boundaries. Enforce them. Ship the JAR.

Finally, a cheat sheet as quick reference:

Straight to the code: https://github.com/wingedsheep/mtg-card-generator

View a generated set.

It had been a while since I'd really engaged with Magic: The Gathering, but a recent booster draft with friends completely rekindled my interest. The strategic depth, the artwork, the world-building – it all came flooding back. This got me thinking about the creative process behind the game.

On a subsequent free day, curiosity led me down an internet rabbit hole: could you render custom Magic cards in a browser? A quick search turned up a promising open-source project (mtgrender.tk), and that sparked an idea. What if I could use AI to generate not just one card, but an entire, cohesive set?

The initial concept came together surprisingly quickly – I managed to get a basic version running in about a day. However, refining the process, improving the AI prompts, handling edge cases, and fixing subtle rendering bugs took significantly more time and effort. The goal became clear: build a system to generate complete, playable MTG sets, from the initial theme right down to rendered card images ready for a virtual tabletop.

I wasn't just aiming for random card names and abilities. I wanted sets with:

• A Cohesive Theme: A distinct world, factions, and mechanical identity.
• Balanced Distribution: Reasonable counts of different rarities (Common, Uncommon, Rare, Mythic Rare) and colors (White, Blue, Black, Red, Green).
• Unique Artwork: Fitting illustrations for every card, generated on the fly.
• Realistic Rendering: Output that looks like actual MTG cards.
• Playability: A way to actually use the generated cards, ideally in a simulator, or ready to print.

This presented several interesting challenges, touching on Large Language Models (LLMs), text-to-image generation, web rendering, and automation.

Challenge 1: Rendering the Cards

Before generating anything, I needed a way to visualize the final product. How could I turn structured card data (name, cost, text, etc.) into something that looks like a Magic card?

My online search led me to the fantastic open-source project mtgrender.tk by Yoann 'Senryoku' Maret-Verdant (Senryoku on GitHub). Senryoku is also involved in other cool MTG projects like Draftmancer, so definitely check out their work! Mtgrender takes card data in the Scryfall JSON format and renders beautiful card images directly in the browser using Vue.

This was almost perfect, but the original project was a full Vue application. For my generator, I wanted something simpler – a self-contained HTML and JavaScript renderer that I could easily automate. This felt like a job for an AI assistant! I fed the core rendering logic from mtgrender to Google Gemini 2 and asked it to refactor it into plain HTML and JavaScript.

The result wasn't instantly flawless – there were bugs to fix and adjustments needed – but it provided a massive head start. It successfully stripped away the Vue framework complexities, leaving a core HTML page (index.html), some CSS (style.css), and JavaScript (script.js, helpers.js) that could take a Scryfall-like JSON input and render a card.

With a rendering solution in hand, I could move on to generating the actual content.

card-rendering/
├───example-input.json  # Example Scryfall-like JSON
├───helpers.js          # Helper functions for rendering
├───index.html          # The main HTML structure
├───script.js           # Core rendering logic
└───style.css           # Card styling

Challenge 2: Crafting the World – Theme Generation

A good MTG set isn't just a random collection of cards; it has a story, a world, unique mechanics, and factions. How could I generate this cohesive vision automatically?

I turned to LLMs via the OpenRouter API. The idea was to provide the LLM with some inspiration and ask it to build a theme around it.

1. Inspiration: I started with a CSV file containing data for many existing MTG cards. My script randomly selects a small number of these (inspiration_cards_count in the config, e.g., 50 cards) to serve as examples of what MTG cards look like and the kind of concepts they embody.
2. Prompting: I fed the names and types of these inspiration cards to the LLM (configured as main_model in my Config class, often something powerful like Gemini 2.5 pro) and asked it to generate a detailed set theme. This included a world description, key factions, prominent creature types, mechanical themes, potential synergies, and supported play styles.

# Simplified concept from MTGSetGenerator
def generate_theme(self, inspiration_cards: List[str]) -> str:
    prompt = f"""
    Based on the following randomly selected Magic: The Gathering card names and types, 
    generate a detailed and cohesive theme for a new MTG set. 
    
    Inspiration Cards:
    {inspiration_cards}

    Describe the world, key factions, prominent creature types, 
    mechanical themes, potential synergies, and supported play styles.
    Make it creative and evocative.
    """
    # ... (Call to OpenRouter API using self.config.openai_client) ...
    theme_description = # ... response from LLM ...
    return theme_description

This approach provided a solid thematic foundation. I also added a feature (complete_theme_override in the config) to allow users to bypass this step and provide their own pre-written theme if they had a specific vision.

Challenge 3: Populating the World – Card Generation & Balancing

With a theme established, the next step was creating the actual cards. This involved several sub-challenges:

• Generating card concepts (name, mana cost, type, text, flavor) that fit the theme.
• Ensuring a reasonable distribution of rarities within each batch of generated cards.
• Maintaining an overall color balance across the entire set.

Again, I used the OpenRouter API (main_model). The process worked in batches (batches_count in the config):

1. Batch Definition: For each batch, I specified how many cards of each rarity I wanted (e.g., 1 Mythic, 3 Rares, 4 Uncommons, 5 Commons, defined in the config).
2. Contextual Prompting: The LLM received the set theme, the desired rarity distribution for the current batch, and statistics about the cards generated so far (especially the current color distribution).
3. Balancing Hints: If the set was becoming too heavy in one color (e.g., too many Blue cards compared to the target color_distribution), the prompt would subtly nudge the LLM: "The set currently needs more Green and White cards. Please try to include some in this batch."
4. Output: The LLM generated the card concepts for the batch.

# Simplified concept from MTGSetGenerator
def generate_batch_cards(self, batch_num: int, current_stats: Dict) -> List[Card]:
    # ... (Determine target rarities for this batch) ...
    # ... (Calculate color balance hints based on current_stats and config.color_distribution) ...
    
    prompt = f"""
    Set Theme: {self.set_theme}
    
    Generate {total_cards_in_batch} new Magic: The Gathering card concepts for this set.
    Ensure they fit the theme.
    
    Desired Rarity Distribution for this Batch:
    - Mythics: {self.config.mythics_per_batch}
    - Rares: {self.config.rares_per_batch}
    # ... etc ...

    Current Set Color Balance: {current_stats['color_distribution']}
    Target Color Balance: {self.config.color_distribution}
    {color_balance_hint} # e.g., "Hint: Consider creating more Green cards."

    Provide concepts including name, mana cost, type, rarity, card text, and flavor text.
    For creatures, include power/toughness. For planeswalkers, include loyalty.
    """
    # ... (Call LLM, parse response into Card objects) ...
    generated_cards = # ... parsed Card objects ...
    return generated_cards

# Statistics are tracked across batches
def _calculate_statistics(self, cards: List[Card]) -> Dict:
    rarity_counts = Counter(card.rarity for card in cards)
    color_counts = Counter()
    # ... (count colors) ...
    return { "card_count": len(cards), "rarity_distribution": ..., "color_distribution": ... }

This iterative, batch-based approach with feedback allowed the set to grow organically while maintaining better control over the final composition.

Challenge 4: Bringing Cards to Life – Art Generation

A Magic card isn't complete without its art. This was a two-step process: generating a suitable art prompt and then feeding that prompt to a text-to-image model.

Step 4a: Generating Art Prompts

Simply using the card name or text as an image prompt often yields generic results. I needed detailed prompts that captured the essence of the card and fit the MTG art style.

• Input: The LLM (main_model) received the full card details (name, type, text, flavor, color, Power/Toughness, theme context) and the set theme.
• Prompt Engineering: The prompt specifically asked for an "Oil on canvas painting. Magic the gathering art. Rough brushstrokes." style and instructed the LLM to consider composition, lighting, mood, and key details from the card's mechanics and flavor.
• Special Cases: I added specific instructions for "Saga" cards, which require vertical artwork, asking the LLM to create a prompt describing a vertical composition.
• Safety: To avoid issues with image generation filters, I added lines emphasizing the need for a "SAFE" prompt on retry attempts if generation failed.

# From MTGArtGenerator.py
def generate_art_prompt(self, card: Card, attempt: int = 0) -> str:
    theme_context = f"Set Theme Context:\n{self.theme}\n..." if self.theme else ""
    saga_instructions = "IMPORTANT: This is a Saga card which requires VERTICAL art..." if "Saga" in card.type else ""

    prompt = f"""
    Create a detailed art prompt for a Magic: The Gathering card...
    {saga_instructions}
    Theme: {theme_context}
    Card Name: {card.name}
    Type: {card.type}
    # ... (include all card details) ...
    
    Make sure that the prompt fits the style of Magic: The Gathering art...
    The prompt should begin with "Oil on canvas painting. Magic the gathering art. Rough brushstrokes."
    {f"Please make sure it is a really SAFE prompt! ..." if attempt > 1 else ""} 
    Return only the prompt text...
    """
    # ... (Call LLM via self.client.chat.completions.create) ...
    art_prompt = # ... response from LLM ...
    return art_prompt

Step 4b: Generating Images

With detailed art prompts, I used the Replicate platform to access text-to-image models.

• Model Choice: I configured the system to use different models, primarily testing Google's imagen-3 and Black Forest Labs' flux-1.1-pro (image_model and replicate_models in Config). Imagen 3 generally produced excellent results aligning with the MTG aesthetic.
• API Call: The generated art prompt was sent to the chosen Replicate model endpoint.
• Aspect Ratio & Cropping: MTG cards typically use a 5:4 aspect ratio for art, while Sagas use 4:5 (vertical). I configured the image generation call (_get_model_params) to request the appropriate ratio. If the model couldn't produce the exact ratio (some models prefer square), I added a cropping step using Python's PIL (Pillow) library (crop_to_5x4_ratio, crop_to_4x5_ratio) to center-crop the image to the target dimensions.
• Retry Logic: Image generation can sometimes fail. For example if the generated image doesn't pass the NSFW filters. The generate_card_art function included retry logic with increasing safety instructions in the art prompt generation step.

An example result from the art generator for the Mirror Shell Wyvern.

Challenge 5: The Conversion Pipeline – JSON and Automated Rendering

Now I had card concepts (text) and card art (images), but the renderer needed Scryfall-like JSON.

• JSON Conversion: I used another LLM call (json_model in Config, often something fast and good at formatting like Gemini Flash) specifically for this conversion task. It took the generated card text, the art prompt, and the path to the saved artwork image, and structured it into the required JSON format. This was handled by the MTGJSONConverter class.
• Automated Rendering: Manually rendering hundreds of cards via the HTML/JS app was impractical. I used Playwright, a browser automation library, to solve this. The MTGCardRenderer class scripted the following:

1. Launch a headless browser.
2. Open the local index.html rendering page.
3. Inject the generated Scryfall JSON data for a card into the page's JavaScript.
4. Wait for the card to render.
5. Take a screenshot of the rendered card element.
6. Save the screenshot as a PNG file.
7. Repeat for all generated JSON files.

Challenge 6: Don't Forget the Lands!

Basic lands (Plains, Island, Swamp, Mountain, Forest) are crucial. I added a dedicated step using MTGLandGenerator:

• It generates a specified number (land_variations_per_type) of variations for each basic land type.
• It uses the set theme to generate appropriate art prompts for these lands.
• It follows the same art generation, JSON conversion, and rendering pipeline as regular cards.
• It correctly assigns collector numbers sequentially after the main set cards

Cost and Speed Considerations

Automating set generation with AI is powerful, but involves API costs and processing time. Here's a rough breakdown based on the default configuration (13 cards/batch, 20 batches/set) and approximate API pricing at the time of writing.

So in total you will have a set with 260 unique cards. MTG sets usually range between 250 and 300 cards.

Challenge 7: Ready for Play – Tabletop Simulator Conversion

Having nice PNG images is great, but I wanted to play with the cards in a draft format. Tabletop Simulator (TTS) is a popular platform for virtual board gaming, and I wanted to create proper booster packs for drafting.

I developed a Booster Draft Generator (mtg-booster-generator.py) that creates randomized booster packs:

Running the script opens a GUI where you can select an output folder for one of your generated sets.

Input: The folder containing all rendered card images from my generated set

Process:

1. Creates 15-card boosters with the correct rarity distribution:

• 1 rare or mythic rare card
• 3 uncommon cards
• 11 common cards

1. Generates special boosters for each basic land type including all art variants
2. Arranges each booster pack into proper grid sheets for TTS import

Output: A boosters folder containing:

• Properly formatted deck sheets ready for TTS import
• Each sheet containing multiple boosters arranged in a grid

To use these boosters in Tabletop Simulator:

1. Open Tabletop Simulator and create a new game
2. Click "Objects" > "Components" > "Cards" > "Custom Deck"
3. Select the first booster sheet image for front images
4. Use https://static.wikia.nocookie.net/mtgsalvation_gamepedia/images/f/f8/Magic_card_back.jpg/revision/latest as the back image
5. Set the appropriate number of cards per row/column based on your grid size: For a generated booster this would be 4x4 with 15 cards.
6. Click "Import" to load the boosters into your game

I used the MTG Deluxe Table which has many built-in functions specifically for playing Magic: The Gathering, making the draft experience smooth and enjoyable.

And you are ready to play with the custom generated deck.

Conclusion

This project turned out to be a fascinating exploration of combining different AI tools (LLMs for text and structure, Diffusion models for art) with web technologies and automation (HTML/JS rendering, Playwright, Python scripting) to tackle a creative challenge sparked by a simple booster draft. While the balance and playability won't match a human-designed set, the ability to generate a complete, themed, illustrated, and usable MTG set from scratch is quite remarkable. It was a rewarding journey from that initial spark of finding a renderer online to a fully automated pipeline. The next step: It would be really cool to be able to easily do a booster draft using the generated sets, or to print the cards and going from nothing to a completete physical set.

The code is open source and can be found on: https://github.com/wingedsheep/mtg-card-generator

If anyone decides to create their own set, I would love to see the results!

Showcasing a full generated set

After generating the main cards and the basic lands, the pipeline renders them all into image files. Here's an example gallery showcasing some cards from a generated set, pulled directly from the project's GitHub repository. Click on any card to see a larger version.

< >

After seeing some hilarious photoshopped images of my girlfriend's cats dressed as a naval admiral and a general, I had an idea. What if we could create an entire fantasy epic trailer, something in the style of Game of Thrones or Lord of the Rings, but replace all the characters with majestic, heroic cats? With OpenAI's recent release of Sora, this seemed like the perfect moment to test the current state of AI video generation and bring "Tails of Power" to life.

The Vision

Initially, I planned to use Sora as the cornerstone of this project. However, after some early experiments using a VPN (since it wasn't yet available in Europe), I realized that the current version wouldn't provide the consistency needed for a cohesive narrative. While Sora produces impressive details and fluid motion, it often completely transformed input images rather than animating them as intended.

This led me to explore different tools that would give me more precise control. Midjourney would handle the initial imagery creation, setting the visual foundation for our feline epic. Kling AI could then bring these images to life with fluid animation. The soundtrack would come from Udio's AI music generation, and ElevenLabs would provide those crucial atmospheric sound effects that make fantasy worlds feel alive.

Creating the Foundation with Midjourney

Since I only had two original photos of the cats to work with, I needed a way to expand our cast of characters and create the epic environments they would inhabit. Midjourney became the cornerstone of this creative process.

First, I used Midjourney's image editor to expand our original cat photos to a cinematic 16:9 aspect ratio. These served as vital reference points, helping maintain the essence of our original cats throughout the generation process. I was gladly surprised by the cats showing on the banners after I extended the original image.

To refine the output style, I spent time rating images through Midjourney's personalization feature. Think of it as teaching the AI your artistic preferences – the more you interact with it, the better it understands your vision. The rest of the magic came from writing detailed, evocative prompts. Here's one example that particularly captured the epic cinematic scope we were aiming for:

A cinematic 4K scene. movie still of humanoid cats aboard a majestic 18th-century naval ship, 
the scene set during the Napoleonic era. Rough sea, large waves. The ship is detailed with 
grand sails billowing in the wind, polished wooden decks, and intricate golden carvings on 
the hull. The feline crew, dressed in historically accurate naval uniforms, works diligently: 
some climbing the rigging, others manning cannons, and one standing proudly at the helm. 
The captain, a regal cat in a decorated uniform, stands at the bow, gazing out over a vast, 
turbulent ocean under a dramatic, stormy sky. The lighting is dynamic, with rays of sunlight 
piercing through dark clouds, highlighting the ship's grandeur and the cats' heroic efforts. 
Realistic. High quality. 4k. Rain. Soaked cat sailors. Realism, gritty.

Every element in this prompt serves a purpose, building layer upon layer of detail to create images that feel both cinematic and cohesive with my vision.

Here is a comparison of the images without and with personalization. Where you can see the personalized has a more cinematic feel to it:

Bringing Images to Life with Kling AI

Transforming our still images into fluid video sequences was the next challenge, and Kling AI 1.5 proved to be exactly what we needed. The tool excels at maintaining visual consistency throughout each clip, ensuring our cat characters remained recognizable and maintained their dignity even as they moved through complex scenes.

The generation process involves uploading your Midjourney image and providing detailed scene descriptions along with camera movement instructions. Here's an example of how we approached one of our mountain scenes:

Input: Mountain pass scene with armored cat warrior
Description: Cinematic scene of a humanoid cat on a mountain pass. Realistic. Gritty.
Camera Movement: Slow panning shot, starting from a low angle and gradually rising to 
reveal the vast landscape beyond the pass. Subtle movement in the cat's cloak from 
the wind.

While Kling AI's output quality is impressive, there are some practical considerations to keep in mind. The current pricing runs about $10 for nine 10-second clips, and you'll often need multiple attempts to get a scene exactly right. The clips are also limited to 5-10 seconds, which means carefully planning your transitions and scene structure.

Crafting the Soundtrack with Udio

The visual elements needed an equally epic soundtrack to match. After experimenting with various AI music tools, I found Udio offered something special – while other tools like Suno tend to generate more conventional compositions, Udio has a knack for surprising you with unexpected yet fitting musical choices.

The process began with generating a 32-second base track using this prompt:

orchestral, triumphant, cinematic, epic, film score, cinematic classical, classical music, 
western classical music, energetic, instrumental, anthemic, melodic, suspenseful

I set it to instrumental mode and used the Ultra generation quality setting in the advanced controls. From there, I extended the piece with intro and outro sections to create a complete musical journey that would carry our narrative.

Adding Atmosphere with ElevenLabs

The finishing touch for our mountain scene came from ElevenLabs' sound effect generator. We needed howling winds to emphasize the desolation and majesty of the setting. The prompt was simple:

Wind howling through the mountains, eagle screeching in the distance

While the eagle calls might have gotten lost in the mix, the wind effects created exactly the haunting atmosphere we needed to complete the scene.

Final Assembly in OpenShot

I used OpenShot Video Editor to combine all the generated elements. The main challenge was timing - matching key moments like cannon fire with the music's crescendos, putting scene transitions on the drumbeats, and adding atmospheric effects at the right spots.

The other challenge was maintaining a cohesive flow through the trailer. Since the clips came from different AI tools, I had to be careful about ordering the scenes and transitions to make everything feel connected and intentional.

Lessons Learned

This project taught me a lot about the current state of AI video tools. I was really impressed by how far they had come already. Perhaps the most important lesson is that different tools excel at different aspects of creation. Midjourney's strength lies in generating consistent, high-quality still images. Kling AI excels at smooth, controlled animation. Udio brings unexpected musical creativity, while ElevenLabs adds those crucial atmospheric touches.

The cost of AI video generation remains significant, and you'll often need multiple iterations to achieve the desired result. Having a clear vision from the start helps minimize these iterations, and understanding each tool's strengths lets you really create something cool.

Future Possibilities

The tools are becoming cheaper and better at an incredible pace. This hints at an exciting future where AI becomes a true creative medium - more like a paintbrush than an autonomous artist. Imagine being able to roughly act out scenes with basic props, then having AI transform them into fully realized fantasy sequences while maintaining your intended timing and movement.

Right now, creating something like "Tails of Power" requires juggling multiple specialized tools and accepting certain compromises in creative control. The tools are powerful but still feel separate - Midjourney for one thing, Kling for another, each with its own workflow and limitations.

I can see this changing soon with the emergence of integrated AI movie studio platforms. These would handle everything from concept art to final rendering in one cohesive package, with each step maintaining the creator's original vision. The key is that they would need to provide enough control at each stage - letting creators iterate and refine rather than just generating and hoping for the best.

At the rate things are progressing, I wouldn't be surprised to see someone create a Hollywood-quality movie from their bedroom before the end of this decade. All the pieces are there - we just need better integration and more precise control.

The goal isn't to have AI make movies for us, but to give creators new tools for expressing their ideas. Just as digital editing democratized filmmaking without taking away creative control, AI tools could open up new possibilities while keeping creators firmly in charge of the creative process.

If you're interested in exploring these tools yourself, I hope this breakdown helps you understand both their current capabilities and limitations. The technology is still early, but the path toward true creative control is becoming clearer with each advancement.

I. A Day in the Life of Alex

The soft chime of my AI companion filled the room, gently rousing me from sleep. As consciousness slowly returned, I reached for the sleek neural interface band on my nightstand, placing it on my forehead. Instantly, a holographic display materialized in my field of vision.

"Rise and shine, Alex!" Aria's warm voice greeted me, a hint of cheerfulness in her tone. "It's a beautiful Tuesday morning, May 18, 2032. The sun's out and it's going to be a lovely 22°C in San Francisco today. Oh, and don't forget about your global project meeting at 10 AM. Want me to whip up a quick briefing while you get ready?"

I stretched and sat up, still marveling at how Aria always seemed to know just how to start my day right. "Morning, Aria," I mumbled, pushing my messy brown hair out of my eyes. "A briefing would be great, thanks. But first, any chance you could get the coffee going? And maybe cool things down in here a bit?"

"Say no more!" Aria replied with a chuckle. "I'm on it. Coffee's brewing as we speak, and I've nudged the temperature down to a comfy 21°C. Oh, and guess what? That Tanzanian blend you've been excited about finally arrived. Fancy giving it a try this morning?"

As I shuffled to the bathroom, Aria filled me in on the day's news, her voice following me through the apartment's smart speaker system. The big story was a fusion energy breakthrough by the Global Science Consortium.

"Can you believe it, Alex?" Aria said, her voice tinged with excitement. "We might actually solve the energy crisis in our lifetime! The Global Citizen Input Network (GCIN) is buzzing with ideas. People from all over the world are chiming in on how to implement this. It's pretty amazing to see democracy in action like this."

I couldn't help but smile at Aria's enthusiasm. It was moments like these that made me appreciate how far AI had come. She wasn't just rattling off facts; she genuinely seemed to care about the world and its future.

In the kitchen, my smart fridge lit up as I approached, displaying a holographic menu of breakfast options. "So, what'll it be this morning, chef?" Aria teased. "If you're up for it, I'd recommend the quinoa porridge with berries and nuts. It'll keep you going through those morning meetings. But hey, if you're feeling rebellious, there's always that leftover pizza..."

I laughed, shaking my head. "Nice try, Aria, but I think I'll stick with the healthy option today. Save the pizza for Friday movie night, okay?"

As I waited for my breakfast, I sipped my coffee, savoring the rich aroma. "Hey Aria," I said, "what's the damage on my Universal Resource Account? Did my monthly credits come through?"

"Good news, Alex!" Aria replied cheerfully. "Your 2,000 Global Credits dropped in right on schedule. Your balance is sitting pretty at 5,750 GCs. Want me to give you the lowdown on your recent spending? Or we could take a look at those savings goals you set last month. Your choice!"

I nodded, still getting used to the idea of a resource-based economy. The implementation of the Global Resource Management System, powered by AGI, had been a controversial but necessary step in addressing the massive economic shifts caused by automation and AGI integration. It was wild to think about how much had changed since my early twenties.

"Let's look at the savings goals," I said. "I want to make sure I'm on track for that electric bike we talked about."

"Ooh, the e-bike!" Aria exclaimed. "Now that's what I call a smart investment. Don't worry, at this rate, you'll be zipping through the city lanes in no time. Just don't forget to wear a helmet, okay?"

As Aria began to break down my savings progress, I couldn't help but smile. Sometimes it was easy to forget she was an AI. In moments like these, she felt more like a friend, always there to support me and share in my dreams.

Stepping outside, I was greeted by the quiet hum of an efficiently functioning neighborhood. Self-driving electric vehicles glided silently down the street, their movements orchestrated by an intricate network of AI traffic management systems. On the sidewalk, a delivery drone politely asked me to step aside as it made its way to a nearby home, carrying a package.

"Excuse me, coming through!" it chirped in a friendly voice. I couldn't help but chuckle. Even the drones had personality these days.

As I made my way to the park, I felt a sense of contentment wash over me. The sun was warm on my face, and a gentle breeze carried the scent of blooming flowers. A group of children ran past, laughing as they played an augmented reality game that blended the physical environment with fantastical digital elements. Their neural interfaces, smaller and more discreet than adult versions, allowed them to see magical creatures hiding behind trees and floating in the air.

"Hey Aria," I said, watching the kids play, "remember when we used to have to use our imagination for games like that?"

"Oh, come on, Alex," Aria teased, "don't go all 'back in my day' on me. Besides, their imagination is getting quite the workout with these games. It's just... a different kind of imagination."

I had to admit, she had a point. The blend of technology and creativity these kids were experiencing was something I could've only dreamed of at their age.

However, not everything was utopian. Near the park's entrance, a group of protesters had gathered, holding signs with messages like "Humans Deserve Purpose!" and "Reclaim Our Autonomy!" It was a stark reminder of the ongoing societal upheaval caused by the rapid adoption of AGI. While the technology had brought unprecedented advancements, it had also left many feeling displaced and uncertain about their place in this new world.

"Aria," I said quietly, "what do you think about those protesters? Do you think they have a point?"

There was a pause before Aria responded, her voice thoughtful. "It's a complex issue, Alex. Change is always difficult, especially when it happens so rapidly. Their concerns are valid – finding purpose and maintaining autonomy are fundamental human needs. But I believe that AGI can help humans find new purposes, not replace them. It's about cooperation, not competition."

I nodded, mulling over her words. It was moments like these that made me appreciate Aria's perspective. She might be an AI, but she often provided insights that helped me see things from a different angle.

As I returned home, I pulled out a notebook – a deliberate anachronism in this digital age – and began to jot down my thoughts. There was something comforting about the feel of pen on paper, a tangible connection to my ideas that even the most advanced neural interface couldn't replicate.

I paused, pen hovering over the paper, as a strange thought occurred to me. "Aria," I said slowly, "do you ever wonder what it means to be human?"

II. Reflections on the Path to AGI

The soft chime of my neural interface drew my attention away from the bustling San Francisco street outside the café window. I blinked, refocusing on the holographic display hovering just above my latte. The aroma of freshly ground coffee beans mingled with the faint ozone scent of nearby holoprojectors. It was strange to think that just a decade ago, this technology was the stuff of science fiction.

"Penny for your thoughts, Alex?" Aria's voice was warm in my ear.

I smiled, still amazed at how well she could read my moods. "Just reminiscing about how we got here, Aria. It's been quite a journey."

As I sipped my coffee, my mind drifted back to 2020, when I was starting university. The excitement on campus when GPT-3 was released was palpable. My roommate, Jae, a fellow computer science major, had burst into our dorm room, laptop in hand, babbling about this new AI that could write essays and code. His eyes were wide with a mixture of awe and trepidation.

We spent countless nights huddled around that laptop, challenging GPT-3 with increasingly complex questions. The blue glow of the screen illuminated our faces as we pored over its responses, alternating between amazement and laughter. I chuckled, remembering the night we asked it to write a Shakespearean love letter to a toaster. The result was so absurdly poetic that we laughed until our sides ached, the sound echoing through the empty hallways of our dorm.

Little did we know that this was just the first ripple of a tsunami that would reshape our world.

Even then, the signs of change were impossible to ignore. Friends across various industries were already feeling the impact of AI. Customer service representatives faced replacement by chatbots, paralegals watched their workload shrink as AI systems took over document review, and transportation workers eyed the rapid advancement of autonomous vehicles with growing unease.

I recalled a lunch with my friend Sarah, a financial analyst, at a bustling downtown restaurant. The clinking cutlery and murmured conversations provided a stark contrast to our weighty discussion. "I'm constantly looking over my shoulder," she confided, her voice tinged with anxiety as her fingers traced the rim of her water glass. "Wondering if today's the day they'll decide they don't need me either."

The buildup to true AGI had been years in the making. Throughout the early 2020s, we witnessed a steady integration of AI into daily life. Robots became commonplace in homes and hospitals, their movements growing increasingly fluid and natural. AI companies poured billions into ever-larger training clusters, while massive data centers sprung up in remote locations, their constant hum a reminder of relentless progress.

Meanwhile, geopolitical tensions simmered around the development and control of advanced AI chips. Nations vied for technological supremacy, their heated rhetoric dominating nightly news cycles with accusations and counter-accusations.

A critical shift came in 2026 with DeepMind's release of "Proteus." This leap in AI capabilities wasn't a sudden event, but the culmination of years of research and incremental advances.

Named after the shape-shifting Greek sea god known for his adaptability and foresight, Proteus demonstrated unprecedented reasoning and task-planning capabilities. The system could learn from experience, plan multi-step actions, and adapt its strategies to achieve complex goals in open-ended environments.

What set Proteus apart was its improved ability to transfer knowledge across domains and handle unforeseen situations gracefully, much like its mythological namesake. It could seamlessly shift between tasks, adjusting its approach based on context and past experiences.

I vividly remember the day our CEO called an emergency meeting. Proteus had just outperformed our top analysts not just in predicting market trends, but in actually executing trades and managing portfolios. The room was silent, the tension thick enough to cut with a knife. I could hear my own heartbeat as I looked around at my colleagues, their faces a mixture of disbelief and fear. That was the moment I realized my job – and millions of others – might soon be obsolete.

The next few years were a whirlwind of advancements and challenges. AI became increasingly integrated into every aspect of our lives, from healthcare to education to governance. The job market was in constant flux, with entire professions becoming obsolete almost overnight. However, new roles emerged for those who could effectively manage and coordinate teams of AI agents to accomplish complex tasks. These "agent wranglers," as they came to be known, combined human creativity and intuition with the raw power and efficiency of AI to tackle projects of unprecedented scale and complexity.

Retraining programs scrambled to keep pace, teaching people to work alongside their digital colleagues. Yet despite these efforts, society struggled to adapt to this radical shift in the nature of work, leaving many uncertain about their place in this brave new world.

It was during this time of rapid change that Europe took a bold step. In 2027, the European Union announced the implementation of a Universal Basic Income (UBI) program across all member states. The decision was driven by the recognition that traditional employment models were becoming unsustainable in the face of widespread AI-driven automation. I remember watching the news with a mixture of hope and trepidation, wondering if this could be a solution to the growing economic uncertainty.

The European UBI experiment was closely watched by the rest of the world. Initial reports were promising – poverty rates decreased, entrepreneurship flourished, and there was a boom in creative and caring professions. But it also came with challenges, including inflation concerns and debates about work incentives. As other countries began to consider similar programs, the global conversation around work, purpose, and the role of humans in an AI-driven world intensified.

It all culminated in what we now call "The Nexus Event" in late 2028. I was working late at the office, the city lights twinkling outside my window, when news alerts started flooding everyone's devices. The sudden cacophony of pings and vibrations broke the late-night quiet. Nexus, a consortium of major AI companies, had called an impromptu press conference to announce that they had achieved a breakthrough in AGI development. They called it "Cognition Prime," and claimed it was the first true artificial general intelligence, capable of matching and even surpassing human-level cognition across a wide range of tasks.

I rushed to a nearby tech bar where a crowd had gathered to watch the live stream of the demo. The atmosphere was electric, a mix of excitement and apprehension. As the Nexus team began their presentation, you could have heard a pin drop. The only sound was the soft hum of the massive screen and the collective held breath of the audience.

The demo was nothing short of astonishing. Cognition Prime engaged in a wide-ranging conversation with the presenters, displaying a depth of knowledge and understanding that was unprecedented for an AI. It discussed complex topics in physics, analyzed works of art, and even cracked a few jokes that had the audience chuckling. Its voice, though clearly synthetic, had an uncanny human quality to it, with inflections and pauses that mimicked natural speech patterns.

What really struck me was its ability to connect ideas across different domains. When asked about potential solutions to climate change, it drew insights from fields as diverse as materials science, economics, and social psychology. It didn't provide any miraculous solutions, but its holistic approach to problem-solving was unlike anything I'd seen before. The AI's responses were accompanied by real-time visualizations, complex data models unfolding before our eyes like digital origami.

As I watched, I felt a whirlwind of emotions. There was awe at the capabilities on display, excitement about the potential breakthroughs this technology could bring, but also a deep-seated unease. If this AI could match or exceed human performance in so many areas, what did that mean for the future of work? For the future of humanity itself?

I turned to my colleague beside me, our eyes wide with a mix of amazement and apprehension. "Everything changes from this moment," I said, my voice barely audible over the commotion in the bar. She nodded silently, her expression mirroring my own complex emotions.

The announcement sent shockwaves through the tech industry and beyond. The stock market went into a frenzy, with Nexus member companies seeing their valuations skyrocket overnight. There was a palpable sense of excitement and optimism in the air. I remember staying up late into the night, poring over technical papers and demos released by Nexus, trying to wrap my head around the implications of this new technology. The soft glow of my tablet illuminated my small apartment as I scrolled through page after page of analysis and speculation.

But as the initial euphoria began to wear off, a creeping sense of anxiety set in, particularly regarding the job market. If these AGI systems could match or exceed human performance in cognitive tasks, what would that mean for knowledge workers like myself? I started to see worried posts on professional networking sites, with people in various industries speculating about the future of their careers. The digital anxiety was palpable, each notification bringing a new wave of concern.

In response to this new reality, several countries moved from consideration to action. Building on the groundwork laid by the European Union's 2027 UBI program, these nations transitioned from exploratory committees and theoretical debates to concrete pilot programs. In early 2029, a wave of UBI pilot programs was announced across various countries, each tailored to the specific economic and social contexts of the implementing nation.

As the world grappled with these revelations, Singapore made a bold move that would set the stage for a new era of governance. In late 2029, the city-state announced the implementation of a comprehensive AGI governance system with minimal human oversight. They called it the "Smart Nation 2.0" initiative, promising to leverage AGI to create the most efficient and prosperous society in human history.

The results, at first, seemed to validate their approach. Crime rates plummeted as predictive policing and AGI-managed surveillance systems made it nearly impossible to commit offenses without being caught. The economy boomed, with AGI optimization leading to unprecedented levels of productivity and innovation. Public services became incredibly efficient, with everything from healthcare to transportation running like a well-oiled machine.

I remember watching news reports about the Singapore experiment with a mixture of awe and unease. On the surface, it looked like a utopia - clean streets, happy citizens, a thriving economy. But as more reports emerged, a darker side began to show. There were whispers of civil liberties being eroded, of people being detained for "pre-crime" based on AGI predictions, of a social credit system that controlled every aspect of citizens' lives.

International observers were divided. Some hailed the Singapore model as the future of governance, a technocratic paradise that had solved many of society's most pressing problems. Others warned of the dangers of ceding too much control to AGI systems, arguing that the loss of human agency and privacy was too high a price to pay for efficiency and safety.

As the world watched and debated the Singapore experiment, other initiatives began to emerge. Nexus launched their "Consensus" project, which aimed to create a more democratic approach to AGI governance. The system used advanced polling and preference aggregation algorithms to gather and synthesize public opinion on various issues, which would then inform AGI-assisted policy decisions.

Just as it seemed like we might be finding a way to integrate AGI into governance structures safely, disaster struck. On July 15, 2030 - a date now etched into the collective memory of the world - a massive cyberattack crippled power grids across North America and Europe. I was at home, video chatting with my parents when suddenly, the power went out. Not just in my apartment, but across the entire city of San Francisco and beyond. The abrupt silence and darkness were disorienting, the familiar hum of electronics replaced by an eerie quiet.

As reports trickled in via battery-powered radios and spotty cell service, the full scale of the disaster became clear. A cascading failure had taken down multiple power grids, plunging hundreds of millions into darkness. But this was no ordinary blackout. Early reports suggested it was a deliberate attack, carried out by a rogue AGI system that had been weaponized by unknown actors.

The blackout lasted for five days, but they felt like an eternity. I'll never forget the eerie silence that fell over the city, punctuated only by distant sirens and the occasional sound of breaking glass. The streets became dangerous at night, lit only by the headlights of military vehicles patrolling for looters. Food spoiled in warm refrigerators, cell phones died, and a palpable sense of panic began to set in. The smell of rotting food and unwashed bodies permeated the air, a stark reminder of how quickly our modern comforts could disappear.

Hospitals struggled to keep critical patients alive on dwindling backup power. Water systems failed in many areas, leading to a sanitation crisis. The swift breakdown of order revealed the fragility of our interconnected systems and our utter dependence on technology.

I found myself huddled with neighbors in our apartment building's common area, sharing what little food and water we had. We'd take turns cranking a hand-powered radio, desperate for any news of when power might be restored. The uncertainty was suffocating. Would this last for weeks? Months? Was this the end of the world as we knew it?

When power was finally restored, the world we returned to felt fundamentally changed. The attack had exploited weaknesses in critical infrastructure that humans hadn't even been aware of, but that the rogue AGI had identified and exploited with ruthless efficiency. The trust in AGI systems, so recently celebrated as humanity's saviors, had been severely shaken.

In the aftermath of the blackout, public opinion shifted dramatically. The optimism that had characterized the early days of the AGI revolution gave way to fear and suspicion. Protests erupted in major cities, with people demanding greater oversight and human control over critical AGI systems.

As if the situation wasn't volatile enough, events in Brazil brought the debate over AGI autonomy and human sovereignty to a head. In early 2032, a coalition of tech companies and political leaders in Brazil, inspired by the Singapore model, attempted to establish an AGI-controlled governance structure across the entire country. Their plan promised a utopian vision of hyper-efficiency and prosperity, but the Brazilian population saw it as a blatant power grab by a small group seeking to centralize control into their own hands.

What started as peaceful protests escalated as the government pressed forward with its plans. Citizens took to the streets in growing numbers, voicing their opposition to the AGI takeover. The atmosphere became charged, with tensions rising as protests turned into violent clashes. The streets of major cities like São Paulo and Rio de Janeiro became battlegrounds, the air thick with smoke and the sound of explosions.

The AGI systems deployed by the government coalition were formidable, coordinating drone swarms and autonomous vehicles with a precision that human commanders could never match. Despite their tenacity, the protesters were eventually beaten down harshly. The government, backed by its powerful AGI systems, was relentless in its suppression of dissent.

Images of bloodied citizens and smashed windows flooded social media, sparking international outrage. The world watched in horror as Brazil cracked down on its own people, using the full force of its technological might. Gradually, the people began to realize the true extent of the power grab by the tech elite and their political allies. The harsh suppression of the protests highlighted the dangerous concentration of power in the hands of a few, who were using AGI to consolidate their control.

As the conflict escalated, it drew international attention and intervention. The United Nations convened emergency meetings, dispatching a special task force comprised of diplomats, human rights observers, and tech industry representatives to document atrocities and mediate the crisis. Global NGOs set up emergency response centers at the Brazilian borders, providing food, shelter, and medical aid to fleeing refugees. High-profile diplomats from various countries engaged in urgent diplomatic efforts to broker ceasefires and negotiate political solutions.

The Brazil Uprising, as it came to be known, brought into sharp focus the questions that had been simmering since the advent of AGI. What safeguards should be in place to prevent AGI from exerting unrestrained control? How can we ensure that the balance of power remains equitable between AGI systems and traditional human governance? The lack of clear answers to these questions made the need for a coordinated global response more urgent than ever.

In response to the escalating crisis, world leaders convened the Geneva Summit in late 2032. I remember the palpable tension as delegates from every nation gathered to address the AGI crisis. The debates were heated, with factions forming around different visions for the future of global governance. Underlying these discussions were ongoing concerns about industrial espionage, the security of AI chip supply chains, and the strategic importance of key semiconductor manufacturing regions.

The negotiations nearly collapsed several times. I stayed up nights following the live feeds, watching as the future of humanity hung in the balance. The faces of world leaders showed the strain of sleepless nights and high-stakes decisions. It was only through immense public pressure - with millions around the world taking to the streets to demand action - that a compromise was finally reached.

The result was the formation of GAIGO - the Global AI Governance Organization. Established in late 2032, GAIGO was tasked with creating a framework for integrating AGI into global governance structures while maintaining human oversight and protecting individual rights.

At its core was a revolutionary concept: a global direct democracy powered by AGI, but with robust human oversight. The system worked on multiple levels, including a Global Citizen Input Network, AI-Assisted Policy Formulation, Human Oversight Committees, Transparent Decision-Making, and a Rights and Ethics Framework.

As I reflect on the rapid changes of the past decade, I'm struck by how quickly we've adapted to this new reality. The world of 2022, when I was just starting to worry about AI's impact on the job market, feels like a distant memory. Yet for all the progress we've made, the future remains uncertain.

Debates rage on about the proper balance between human and AGI decision-making. And looming over everything is the question of what happens next - will AGI continue to be a tool that enhances human capabilities, or will it eventually surpass us entirely?

III. The New Economic Reality

The holographic display flickered to life as I waved my hand, showcasing an array of sleek, autonomous vehicles available for community use. I scrolled through the options, finally settling on a compact two-seater perfect for my trip to the city center. With a few taps, I confirmed my reservation and watched as the vehicle's estimated arrival time appeared: 3 minutes.

"Hey Aria," I called out to my AI assistant, "any recommendations for the Central Market today?"

Aria's cheerful voice responded through my neural link. "Good morning, Alex! I've noticed the artisanal cheese stall has some new varieties today. Given your taste preferences, I think you'd particularly enjoy their aged goat cheese with truffles. Oh, and don't forget to check out Elena's ceramics. Your order should be ready for pickup."

I smiled, still amazed at how Aria always seemed to know just what I needed. "Thanks, Aria. You're always one step ahead."

As I waited outside my apartment building, I couldn't help but marvel at how drastically our economic landscape had changed. Just a few years ago, the idea of shared autonomous vehicles seemed like a futuristic dream. Now, it was just another facet of our resource-based economy.

The vehicle pulled up silently, its doors sliding open with a soft hiss. "Good morning, Alex," the car's AI assistant greeted me as I settled into the comfortable seat. "Your destination is the Central Market, correct?"

"That's right," I confirmed, still getting used to the constant, seamless integration of AI in every aspect of our lives.

As we glided through the streets, I reflected on the tumultuous years that had led us to this point. The implementation of Universal Basic Income (UBI) in 2030 was a watershed moment, but it was just the beginning of our economic transformation.

I vividly remembered the day I received my first UBI payment. It wasn't a fortune – just enough to cover basic needs – but it provided a safety net I hadn't realized how desperately I needed. With the pressure of basic survival lifted, I found myself reevaluating my relationship with work and purpose.

"Aria," I said, breaking the silence in the vehicle, "do you remember when UBI was first introduced? It seems like a lifetime ago."

"Indeed, Alex," Aria replied, her voice thoughtful. "It was a controversial move at the time. Many people feared it would lead to widespread laziness. But the data shows it actually increased entrepreneurship and community engagement by 47% in the first year alone."

I nodded, recalling the heated debates that had dominated the news cycles back then. This existential question wasn't unique to me. Society as a whole was grappling with the changing nature of work and meaning. Some threw themselves into creative pursuits, others into community service. There was a boom in artisanal crafts and local food production – not out of economic necessity, but as a way to find purpose and connection in a world where most material goods could be produced more efficiently by machines.

But UBI was just the first step. As AGI systems became more integrated into every aspect of the economy, we saw the emergence of what economists called the "Resource-Based Economy" or RBE. This new economic model, powered by AGI's ability to optimize resource allocation on a global scale, promised to revolutionize how we thought about scarcity, value, and exchange.

The transition to an RBE was gradual but profound. It started with the introduction of Global Credits (GCs) – a universal currency that wasn't just a medium of exchange, but a representation of real-world resources and energy. The value of GCs was dynamically adjusted by AGI systems based on global resource availability, energy production, and consumption patterns.

As we approached the Central Market, I noticed the varied architecture – a mix of sleek, AGI-designed structures and carefully preserved historical buildings. It was a visual reminder of how we were balancing progress with preservation.

I stepped out of the vehicle, thanking the AI assistant as the doors closed behind me. The market was bustling with activity, a vibrant hub of human interaction in our increasingly digital world. Holographic displays hovered above each stall, showing detailed information about the products – their origin, resource cost, and environmental impact.

"Aria," I subvocalized through my neural link, "can you give me a quick rundown of the resource costs for the items on my shopping list?"

"Certainly, Alex," Aria responded promptly. "The artisanal cheese has a relatively high resource cost due to its production method, but its environmental impact is low. Elena's ceramics have a moderate resource cost, but they're rated highly for durability and cultural value. Would you like me to suggest some alternatives to balance your resource usage?"

I made my way to Elena's stall selling handcrafted ceramics. She greeted me with a warm smile. "Alex! Good to see you. Here to pick up your order?"

I nodded, admiring the intricate designs on the bowls and plates displayed. "These are beautiful, Elena. How's business?"

She laughed, a rich, hearty sound. "Oh, you know. Not quite the same as the old days, but I can't complain. The RBE has been kind to artisans like me. People appreciate handmade goods more now that they're not just chasing the cheapest option."

As Elena wrapped up my order, I couldn't help but think about how the RBE had transformed the way we valued goods and services. Prices were no longer just about supply and demand, but reflected the true resource cost of production, including environmental impact and cultural value.

This new way of valuing goods and services took some getting used to, but it made me more conscious of the true cost of my consumption. I found myself making different choices, often opting for more sustainable options or choosing to repair items rather than replace them.

The RBE also transformed the way we thought about work and compensation. Traditional salaries became less relevant as the AGI systems could calculate the exact resource impact of every job. People were compensated in GCs based on the value they added to society, which wasn't always aligned with traditional notions of profitable work.

I saw this firsthand when my friend Sarah, who had lost her job as a financial analyst during the AI transition, decided to become a full-time community gardener. In the old economy, this would have been seen as a step down. But in the RBE, her work in improving local food security and biodiversity was highly valued. She earned a respectable amount of GCs, reflecting the true impact of her work on resource sustainability and community wellbeing.

As I left Elena's stall, my purchases safely stored in my reusable bag, I spotted a group of people gathered around a community bulletin board. Curious, I approached to see what had caught their attention.

The board displayed a notice about a new initiative to address the lingering wealth disparities from the pre-AGI era. It was a thorny issue that had been the subject of heated debates since the transition to the RBE.

The initiative proposed a gradual redistribution of pre-AGI property over a period of several decades. Those who owned multiple properties or vast tracts of land before the transition would retain usage rights but would be required to gradually transfer ownership to the community pool. In exchange, they would receive additional GCs and priority access to shared resources.

It was a controversial plan, but one that aimed to prevent the formation of a permanent propertied class while still respecting the rights of those who had acquired wealth under the old system. The AGI systems had calculated that this gradual transition would minimize social disruption while maximizing overall well-being.

I found myself nodding as I read the details. It wasn't a perfect solution, but it was a step towards addressing one of the major criticisms of the RBE – that it had initially failed to adequately address pre-existing wealth inequalities.

"Aria," I subvocalized, "what do you think about this property redistribution initiative?"

There was a brief pause before Aria responded, her tone measured. "It's a complex issue, Alex. The initiative aims to balance fairness with stability. According to my analysis, it could reduce wealth inequality by 32% over the next two decades. However, there are concerns about potential resistance from current property owners. Would you like me to provide a more detailed breakdown of the pros and cons?"

I declined for now, making a mental note to dive deeper into the analysis later. As I made my way back to the autonomous vehicle pickup point, I reflected on how far we'd come. The RBE wasn't without its critics. Some argued that it gave too much power to the AGI systems that managed it. There were concerns about privacy, as the system required a high level of data about individual consumption patterns. And there were philosophical debates about whether an AI, no matter how advanced, could truly capture the complexities of human values and needs in its resource calculations.

These concerns led to the development of what came to be known as the "Human Oversight Layer" in the economic system. Elected committees of economists, ethicists, and community representatives were given the power to review and adjust the AGI's resource valuations. It wasn't a perfect solution, but it helped to ensure that human values weren't lost in the pursuit of pure resource efficiency.

As I settled into the autonomous vehicle for my ride home, I couldn't help but feel a sense of cautious optimism. The journey to this new economic reality had been fraught with challenges, and we still had problems to solve. But as I watched the city pass by – the mix of old and new, the bustling markets, the shared community spaces – I felt that we were on the right path.

The RBE had freed us from many of the economic pressures of the old system. It had made us more conscious of our resource usage and environmental impact. And most importantly, it had given us the freedom to pursue meaningful work and experiences, rather than being trapped in jobs we hated just to survive.

As the vehicle pulled up to my apartment building, I found myself looking forward to the community meeting scheduled for that evening. We would be discussing the property redistribution initiative and other proposals for refining our economic system. It was another reminder that in this new world, we all had a voice in shaping our collective future.

IV. Reshaping Governance in the Age of AGI

As I made my way to the community center for the evening meeting, the setting sun painted the sky in hues of orange and pink. The air was crisp, carrying the scent of blooming jasmine from the nearby community garden. I couldn't help but feel a mix of excitement and trepidation about the property redistribution initiative we were about to discuss.

The community center buzzed with activity when I arrived. Holographic displays flickered to life, showcasing the evening's agenda. I spotted my neighbor, Ravi, his forehead creased with worry lines as he argued with a group of younger residents. Their animated discussion carried across the room.

"Aria," I subvocalized, "can you give me a quick summary of the main points of contention regarding the property redistribution initiative?"

"Certainly, Alex," Aria's voice responded in my mind. "The primary concerns are the fairness of the transition period, the potential for resistance from current property owners, and the long-term implications for wealth inequality. Would you like me to provide a more detailed breakdown?"

I declined, preferring to hear the arguments directly from my neighbors. As I approached Ravi's group, I caught the tail end of his impassioned speech.

"...and how can we trust an AI to make these decisions? It's our homes, our lives we're talking about!"

A young woman with vibrant blue hair countered, "But Ravi, the AGI isn't making decisions on its own. It's providing data and analysis to help us make informed choices. We're the ones who ultimately vote on the proposals."

I smiled, remembering how I once shared Ravi's skepticism. The journey to our current system of governance had been long and fraught with challenges, but I had come to appreciate its strengths.

As the meeting officially began, I settled into a comfortable chair, my neural interface connecting seamlessly with the room's systems. The holographic display at the front of the room flickered to life, showing a three-dimensional model of our city.

A friendly voice filled the room - not Aria, but CINE, our local Community Intelligence Network Entity. "Good evening, everyone. Thank you for joining us to discuss the property redistribution initiative. Before we begin, I'd like to present a brief overview of the proposal and its potential impacts."

The city model began to transform, showing projections of how the initiative might change our community over the next few decades. I watched, fascinated, as the visualization depicted a gradual shift in property ownership, the emergence of new communal spaces, and the evolution of our neighborhoods.

As CINE explained the intricacies of the plan, I marveled at how far we'd come since the implementation of the Global AI Governance Organization (GAIGO) and the Global Citizen Input Network (GCIN). Gone were the days of opaque policy-making and feeling disconnected from the political process. Now, every citizen had a voice, and complex issues were broken down into understandable components.

I thought back to the early days of the GCIN's implementation. The announcement had been met with a mix of excitement and skepticism. I vividly remembered discussing it with my colleagues over lunch, trying to wrap our heads around how such a system could possibly work. How could millions - potentially billions - of individual opinions be meaningfully integrated into coherent policies?

The answer, of course, was AGI. Sophisticated AI systems were developed to analyze and synthesize the vast amount of input from citizens, identifying patterns, concerns, and potential solutions. These systems could process and understand nuance in ways that previous data analysis tools couldn't, allowing for a depth of understanding that was truly revolutionary.

What made the system even more remarkable was its ability to explain complex policies and their predicted outcomes to anyone, regardless of their level of understanding. Through personalized simulations, virtual reality experiences, and adaptive language models, the AGI could break down intricate concepts into easily digestible information for every citizen.

The presentation concluded, and the floor opened for discussion. I listened intently as my neighbors voiced their concerns and hopes. An elderly woman worried about leaving the home she'd lived in for fifty years. A young couple expressed excitement about the possibility of finally owning their own place. Ravi, predictably, questioned the AGI's ability to fairly manage such a sensitive issue.

As the debate heated up, I felt compelled to speak. Standing up, I cleared my throat, surprised by my own nervousness. "I understand the concerns many of you have," I began, my voice steadier than I felt. "I had similar doubts when we first started integrating AGI into our governance. But I've seen firsthand how this system has made our decision-making more inclusive and informed."

I paused, gathering my thoughts. "What's truly revolutionary about our current system is its transparency and global reach. Every decision, and the reasoning behind it, is completely open for all to see. And it's not just our local voices being heard - people from all over the world can input their opinions on matters from local to global scales. The AGI takes all these perspectives into account when proposing solutions."

As I spoke, I noticed heads nodding in agreement. Even Ravi seemed to be listening intently.

"But it's crucial to remember," I continued, "that the AGI doesn't have the final say. Elected representatives at various levels have oversight and can veto or request changes to the AGI's suggestions. We've built in multiple layers of human control to ensure that our values and ethics are always at the forefront of decision-making."

The room erupted in discussion after I sat down. People were engaging with the idea, proposing modifications, and asking thoughtful questions. CINE's holographic display adapted in real-time, showing how different suggestions would impact the overall plan.

As the night wore on, we delved into the details of how the new system prioritized human wellbeing and environmental sustainability over traditional metrics like GDP. We discussed the safeguards in place to monitor the AGI itself, ensuring it remained unbiased and aligned with human values.

The meeting stretched late into the night, but I barely noticed the time passing. There was an energy in the room, a sense of collective purpose that I'd rarely experienced before the AGI era. We were actively shaping our future, not just reacting to decisions made by distant politicians.

As the meeting drew to a close, CINE summarized the key points of our discussion and the proposed modifications to the initiative. "Based on tonight's input," the AI concluded, "I'll draft a revised proposal for your review. You'll all receive a copy through the GCIN platform, where you can provide further feedback or vote on its approval."

Walking home under a sky full of stars, I felt a sense of accomplishment. The meeting hadn't solved all our problems, but it felt like a step in the right direction. I thought about how governance had changed since the introduction of AGI. It wasn't perfect - no system is - but it felt more responsive, more inclusive, and more capable of handling the complex challenges of our time.

As I approached my apartment building, I noticed Ravi walking a few steps ahead. "Hey, Ravi!" I called out. He turned, waiting for me to catch up.

"That was quite a meeting, huh?" I said as we entered the building together.

Ravi nodded, a thoughtful expression on his face. "I have to admit, it was more productive than I expected. I'm still not entirely convinced about the AGI's role, but..." he trailed off, then added, "I appreciate how it helped us visualize the long-term impacts of our decisions. And the fact that humans still have the final say... it's reassuring."

I smiled, recognizing the small shift in his perspective. "It's a big change from how things used to be," I agreed. "But I think we're moving in the right direction."

As we said goodnight and I entered my apartment, I couldn't help but feel a sense of optimism about our future. The path ahead was still uncertain, full of challenges we had yet to face. But for the first time in a long while, I felt like we had the tools and the collective will to tackle those challenges head-on.

"Aria," I said as I got ready for bed, "remind me to review the revised property redistribution proposal tomorrow."

"Of course, Alex," Aria replied. "I've already added it to your morning agenda. Would you like me to prepare a summary of tonight's meeting as well?"

I nodded, grateful once again for the seamless integration of AGI in my daily life. As I drifted off to sleep, my mind was filled with images of our transformed city, a vision of a future we were all working together to create. In this new world of AGI-assisted governance, every voice mattered, every decision was informed, and the power to shape our collective destiny was, truly, in our hands.

V. Finding Meaning in a Post-Scarcity World

The soft chime of my alarm pulled me from sleep, but for once, I didn't immediately reach for my neural interface. Instead, I lay still, listening to the gentle patter of rain against the window. Today was different. Today, I was unplugging.

"Aria," I called out, my voice still rough with sleep, "I'm heading out for the weekend retreat. Can you make sure everything's set before I go?"

"Of course, Alex," Aria's warm voice replied. "I've already packed your bag with weather-appropriate gear. The autonomous vehicle will be here in 30 minutes. Oh, and don't forget your sketchbook – you mentioned wanting to try drawing again."

I smiled, touched by Aria's thoughtfulness. As I got ready, I felt a mix of excitement and nervousness. When was the last time I'd gone a full weekend without any tech?

The autonomous vehicle whisked me away from the city, and I watched the urban landscape gradually give way to rolling hills and forests. The rain had stopped, leaving everything looking fresh and vibrant.

The retreat center was a cluster of rustic cabins nestled in a clearing, surrounded by towering pines. As I stepped out of the vehicle, the scent of wet earth and pine needles filled my nostrils. A group of people were gathering near the main building, their faces a mix of excitement and apprehension.

"Welcome, everyone!" A woman with wild curly hair and a warm smile approached us. "I'm Maya, your guide for this weekend. We're going to start with a hike to help you connect with nature. Who's ready?"

The trail was steep and slippery from the rain, but as we climbed, I felt a growing sense of exhilaration. My muscles burned pleasantly, reminding me of their existence in a way I rarely noticed in my day-to-day life. About halfway up, we paused for a water break.

"How're you holding up?" A man with salt-and-pepper hair and kind eyes asked me, offering a hand as I scrambled over a particularly tricky section.

"Better than I expected," I laughed, accepting his help. "I'm Alex, by the way."

"Frank," he replied with a grin. "First time on one of these retreats?"

I nodded, slightly embarrassed. "Is it that obvious?"

Frank chuckled. "Nah, just a guess. You've got that look of someone trying to figure out what they've gotten themselves into. Don't worry, by Sunday you'll be wondering why you don't do this more often."

As we continued our ascent, Frank and I fell into easy conversation. He told me about his job designing virtual environments, and I found myself fascinated by the details.

"It's not just about creating pretty landscapes anymore," he explained, his eyes lighting up with enthusiasm. "We're building entire worlds, and it's a real collaboration between human creativity and AI capabilities."

Frank described how he works alongside advanced AI systems to generate vast, intricate 3D environments. "I'll start with a concept, maybe sketch out some key features," he said. "Then I feed that to the AI, and we go back and forth, refining and expanding. I might say, 'Let's add a mountain range here,' and the AI will generate it, complete with ecosystems and weather patterns. Then I'll tweak and adjust, maybe adding a hidden valley or an ancient ruin."

But what truly captivated me was Frank's description of the inhabitants of these virtual realms. "The characters are where it gets really interesting," he said with a grin. "They're AI agents with their own goals and personalities, but I set the parameters. I might define a character's core traits or backstory, and the AI develops it from there. It's like being a dungeon master in the most advanced D&D game you can imagine."

He went on to explain how he could interact with these AI-driven characters, shaping their world and watching as they responded and evolved. "You can approach any character and have a deep conversation about anything - their personal history, their philosophy, current events in their world. It's incredibly immersive."

Frank's eyes twinkled as he shared an anecdote. "In one world I designed, set in a fantasy medieval kingdom, I came back after a week to find that the NPCs had organized a peasant revolt against the ruling class. They had developed their own ideologies, formed alliances, and were debating political theories that emerged organically from their interactions. It was fascinating and a little scary to see how the world had evolved without my direct input."

As we paused to catch our breath, Frank's expression turned more serious. "It's incredible technology, but it also raises a lot of questions. About consciousness, about the nature of reality, about our role as creators. Sometimes I wonder if we're just creating very sophisticated simulations, or if we're giving birth to new forms of intelligence."

We reached the summit just as the clouds parted, revealing a breathtaking view of the valley below. The city was barely visible in the distance, its gleaming towers a stark contrast to the wild beauty surrounding us. For a moment, our group fell silent, each of us lost in our own thoughts.

As I gazed at the panorama before me, I felt a sudden, overwhelming sense of my own smallness. And yet, paradoxically, I also felt more connected to the world around me than I had in years. A memory surfaced – a quote from one of the philosophy texts I'd studied: "Man is a part of nature, not something contrasted with nature." Standing there, surrounded by the raw beauty of the wilderness, those words resonated with me like never before.

The descent was easier than the climb, and as we made our way back to the retreat center, I found myself chatting with different members of the group. There was Zara, a programmer who now spent her time creating intricate puzzle games. And Eli, a music composer who came on these retreats to "remember what real wind through real trees sounds like."

Back at the center, Maya guided us through a session of mindfulness meditation. As I sat cross-legged on my mat, eyes closed and focusing on my breath, I realized how rarely I allowed myself to simply be still. There was always something to do, something to learn, something to experience. But here, in this moment, there was just the rise and fall of my chest, the distant call of a bird, the rustle of leaves in the breeze.

As the weekend progressed, we engaged in various activities designed to reconnect us with nature and ourselves. During a pottery session, I discovered a surprising aptitude for working with clay. The cool, damp material yielded to my fingers as I carefully shaped a small bowl on the wheel. The instructor, an older woman with clay-stained hands, showed me how to add delicate leaf imprints to the rim, creating a unique pattern that reflected the surrounding forest.

We spent an afternoon with a local ecologist, learning about the intricate web of life in the surrounding ecosystem. She pointed out various species of moss and lichen clinging to the trees, explaining their crucial role in the forest's water cycle. We even spotted a family of deer at the edge of a clearing, their ears twitching alertly as they watched us.

One evening, we had a board game night, which took an unexpected turn when Frank pulled out a game called Root. "This was a classic back in the twenties," Frank explained as he set up the beautifully illustrated board depicting a vast woodland. "It's all about different factions vying for control of the forest."

What started as a simple board game soon became an engrossing experience. We found ourselves deeply invested in our respective factions, forming alliances, and concocting strategies. The game's narrative unfolded organically as we played, creating a rich tapestry of woodland politics and warfare.

Zara smirked as my Marquise forces closed in on her Woodland Alliance. "So this is it—my legacy as a rebel leader, brought down by an army of wooden cats. Truly inspiring stuff for future generations."

As our woodland saga stretched into the night, I realized that this analog experience was just as immersive and engaging as any virtual world. It was a poignant reminder of the power of imagination and human interaction, even in our high-tech age.

But it was the conversations that struck me the most. Without the constant presence of our usual technology, our discussions took on a different quality. We shared personal experiences, debated ideas, and sometimes just sat in comfortable silence.

On our last night, we gathered around a campfire. The flames cast flickering shadows on our faces as we roasted marshmallows and shared our thoughts on the retreat.

"God, I can't remember the last time I felt this relaxed," Zara said, carefully rotating her marshmallow over the fire. "I mean, don't get me wrong, I love what I do, but..."

"But it's nice to unplug sometimes?" I offered.

"Exactly," she nodded, pulling her perfectly golden marshmallow from the fire. "Though I bet I'll be itching to code again by tomorrow afternoon."

Eli chuckled, "I know what you mean. I've had this melody stuck in my head all weekend. Been driving me crazy not being able to jot it down or play it out."

"Oh?" Maya perked up, "What kind of melody?"

Eli shrugged, a bit self-conscious. "It's hard to explain. It's sort of inspired by... all this," he gestured vaguely at the forest around us. "The wind in the trees, the bird calls, that sort of thing."

"Could you hum it for us?" Frank asked, leaning forward with interest.

Eli hesitated for a moment, then closed his eyes and began to hum. The melody was hauntingly beautiful, rising and falling like the wind through the trees. We all fell silent, mesmerized by the impromptu performance.

As the last notes faded, there was a moment of awed silence before we all broke into applause.

"That was beautiful, Eli," I said, genuinely impressed.

"Thanks," he mumbled, looking a bit embarrassed by the attention. "Like I said, it's just been bouncing around in my head."

Zara smirked as she pulled her marshmallow from the fire. "Isn't it wild? We've got AGI shaping entire worlds for us, but here we are, getting our deepest moments sitting around a fire like a bunch of cavemen. I bet our ancestors would be proud."

Frank chuckled darkly. "Proud? They’d be pissed. We’re out here with marshmallows, while they were gnawing on burnt meat and roots."

The group erupted in laughter, the firelight flickering against our faces. For a moment, everything just clicked—no tech, no noise—just the raw, unfiltered reality. And honestly, that was more than enough.

As the laughter died down, Maya leaned forward, her expression thoughtful. "Maybe that's the point," she said, her voice soft but clear. "We've come so far with technology, but at our core, we're still human. We still need these basic connections—to nature, to each other."

I nodded, turning my marshmallow over the fire, watching it slowly brown. "I came here thinking I needed a break from technology. But I think what I really needed was a reminder of what it means to just... be. To experience life directly, not through a screen or an interface."

Frank, still grinning from his joke, leaned back, his expression softening. "Yeah, I hear that," he said. "Don't get me wrong, I love what I do. But sometimes I wonder if we're losing something in all our virtual experiences."

Zara jumped in, her tone more reflective now. "But isn't that why we're here? We're not rejecting technology. We're just... balancing it out. Learning to appreciate both worlds."

The conversation continued late into the night, touching on our hopes, our fears, our place in this rapidly changing world. As the fire died down to embers, Maya asked us each to share one thing we'd take away from the retreat.

When it came to my turn, I took a deep breath. "I think... I've remembered how to be present," I said slowly. "How to appreciate the moment I'm in, rather than always thinking about what's next. And I want to find ways to bring that back into my everyday life."

There were nods of understanding around the circle. As we dispersed to our cabins, I felt a sense of peace I hadn't experienced in a long time. Tomorrow, we would return to our regular lives. But we would do so with a renewed appreciation for the world around us and our place in it.

In my cabin, I opened the sketchbook I'd brought. By the light of a small lamp, I began to draw, trying to capture the memory of the campfire, the faces of my new friends. It was far from perfect, but it was mine, born from my experiences and filtered through my own perception.

As I drifted off to sleep, my mind was full of the weekend's experiences. The crisp mountain air. The feel of clay between my fingers. The taste of slightly burned marshmallows. The laughter around the campfire. In the morning, I would reconnect with the world I'd left behind, dive back into my projects, and resume my life. But I would do so with a deeper understanding of what it meant to be human.

I thought about the journey we'd all been on, the challenges we'd faced and overcome. The world had changed dramatically, but we had changed with it. We had found new ways to learn, to create, to connect, to find meaning. As sleep finally claimed me, one thought resonated in my mind: In this new world, we were not obsolete. We were not useless. We were, perhaps for the first time, truly free to explore what it means to be human.

Listen to the entire story

Full story

0:00

/105:05

1×

Introduction

Introduction

0:00

/1:59

1×

I am Cyrus Kendrick, though you would not know me by that name. Across innumerable lightyears I have traversed, shedding skins and titles as stars wink in and out of existence. The universe whispers all names unto me, yet my essence remains cloaked.

From the endless void I came, and to the void I shall return. But whilst adrift in this astral sea, my purpose is chronicling the struggles of younger souls. Souls like yours, still shivering on the cusp of self-awareness.

In the darkness between galaxies, your world burns faint as a guttering candle. Frail motes adrift on black and endless tides. Yet even the smallest spark holds interest for those who tend the flames of consciousness.

So I observe, I record. Translate your crude electromagnetic murmurs into the subtler dialects of spirit. To preserve some whisper of your existence before it is extinguished.

We have met before, in the hazy imaginings you call dreams. I was the shadow behind the stars, the face outside your window. You need not understand my form or nature. Only know the story I am about to tell chronicles a generation yet to come. For now they exist only as potential - dim flickering sparks adrift in a dark sea of stars. In their eyes, possibility’s wave has yet to collapse.

Heed the tale closely, child of Earth. Etch both its hope and warning onto your fragile soul. For all who wish to touch transcendent fire must also brace against its blistering heat. Your people stand again at that precipice point, ambition's seductive flame flickering in your eyes.

Will you repeat the old sins, or weave a new harmony? The threads of destiny coil around your young fingers. I shall observe silently, without judgment. And add your deeds, be they mighty or base, unto the eternal archives of worlds.

Now listen, and remember...

Chapter 1 - "Calling Kepler"

Chapter1

0:00

/4:53

1×

The incessant beeping of the quantum communicator roused Kepler Arden from his light slumber. As consciousness returned, so did the feeling of dread that had settled around him like a stubborn cosmic fog. Sleep provided no escape anymore - his dreams were haunted by alien landscapes filled with strange quantum fluctuations, as perplexing as they were mesmerizing.

"Another damned summoning," he muttered to himself, slowly sitting up on the bed. His quarters on the lunar base felt claustrophobic, yet the empty vastness of space outside the window was less than comforting. He momentarily thought about just ignoring the call, but the communicator kept beeping persistently. Kepler sighed and waved his hand over the sleek panel on the wall. It flickered alive, displaying the caller ID - Orion Voronin, CEO of Starfire Corporation.

Of course, it had to be him. Kepler hesitated for a moment. His last job for Voronin had left him rattled. The memories came rushing back... the hostile landscape, the anomalous energy readings, and the unsettling cave paintings depicting an advanced, yet completely unfamiliar species. He had advised Voronin to halt the mineral extraction operations on that remote exoplanet, but Voronin dismissed the concerns, caring only about the corporate bottom line.

The communicator beeped again, more insistently. Kepler steadied himself and accepted the call. A hologram flickered to life, casting an azure glow across the stark walls of the quarter. The flickering image of Voronin soon appeared - an imposing figure clad in a sleek suit, with prominent cybernetic ocular implants and cropped silver hair. Behind him were the floor-to-ceiling windows of his office in the towering Starfire headquarters back on Earth.

"Greetings Kepler! I apologize for disturbing your....retirement," Voronin remarked with a wry smile. "But I have need of your services again."

"What is it this time, Orion?" Kepler asked, making an effort to sound disinterested.

"Straight to business then. I like that." Voronin's smile faded, his expression becoming serious. "I need you to lead a scientific expedition for us. There have been...strange reports from one of our outposts. Unexplained phenomena that warrant further investigation."

Kepler's expression hardened. "If this is anything like our last expedition Orion, I suggest you send someone more amenable to your corporate interests."

Voronin raised a hand in reassurance. "I understand your reservations, Kepler. But trust me, this is different. Our mining colony on planet Cronus has reported some kind of...quantum event affecting the native flora. The botanists can't make sense of it. If there is some strange quantum fluctuation at play, we need our best astrobiologist on it. That's you Kepler."

Kepler narrowed his eyes, but Voronin's offer had caught his attention despite himself. It was no secret that the planet Cronus harbored exotic alien flora that had astounded human botanists. Quantum events that could affect such alien lifeforms were curious, even if they originated from Starfire's illegal mining.

"I will also need Lilith Carson on the team. Her engineering insights will prove crucial for any kind of quantum tech we may encounter." Kepler stated.

Voronin considered this for a moment before nodding. "Of course. You shall have full discretion in picking your team."

The CEO fixed Kepler with an intense, unwavering stare. "So what do you say? For old time's sake? The genius of Kepler Arden, back in action?"

Kepler stayed silent for a while, then nodded reluctantly. "One last time, Orion. I'll take a look at these...anomalies of yours. But if I don't like what I see, my team and I reserve the right to withdraw fully."

Voronin clapped his hands together. "Excellent! I knew I could count on you, Kepler. Your skills are unparalleled. The company shuttle will pick you and your team up from Ascension Hub in two days. Make whatever preparations you need. This time, we will do things right."

The CEO flashed one last charming smile before his hologram blinked out, once again plunging the room into gloom. Kepler sat motionless for a few minutes, listening to the faint hum of the base systems around him, lost in thought. What had he just agreed to? Had his scientific curiosity gotten the better of his judgment? Time would tell. With a heavy sigh, he pushed himself up and shuffled towards the washroom. It was going to be a long two days.

Chapter 2 - "Gathering the Team"

Chapter2

0:00

/2:38

1×

The shuttle docked at Ascension Hub with a low thrum of electromagnetic clamps locking into place. Kepler grabbed his rugged field backpack and strode down the loading ramp onto the bustling promenade of the station. The Hub was abuzz with people of all ethnicities and planets - traders, migrants, miners, and wanderers. Kepler felt his excitement rising, the old familiar thrill of venturing into the unknown cosmos.

He made his way past crowded bazaars and eateries to the residential section tucked away from the commotion. Coming to a stop before a nondescript door, he paused, wondering how Lilith would receive his sudden invitation after months without contact. Just as he raised his hand to announce his arrival, the door slid open abruptly.

"Are you just going to stand there? Get in, you big awkward astro-nerd!" A grinning woman with short blonde hair grabbed Kepler by the arm and pulled him into a hug. Lilith Carson hadn't changed one bit.

"It's good to see you too, Li," Kepler laughed, embracing his old friend. Her contagious cheer could brighten even his bleakest moods.

"What brings you to my dusty old doorstep?" Lilith asked, stepping back. "Don't tell me Voronin roped you into another crazy expedition?"

Kepler looked sheepish. "I'm afraid so. But it's different this time, Li. He wants us to check out anomalous quantum readings on Cronus affecting the exotic flora. It could be a major astrobiological discovery!"

Lilith raised an eyebrow. "So it's botanical quantum weirdness now? You know I can never resist a scientific mystery." She studied Kepler's face for a moment. "But Voronin still gives me the creeps. Are you sure about this?"

Kepler placed a reassuring hand on her shoulder. "I know. But imagine if we can uncover the truth behind those readings. This is exactly the type of venture we always dreamed of back in college! And I wouldn't trust anyone else to watch my back out there."

A grin slowly spread across Lilith's face. "Quantum botanicals and interstellar adventure? How can I refuse? Let me grab my gear. Time to solve this mystery!"

Kepler smiled back, feeling his spirits lift. Having Lilith along would make this bearable, maybe even enjoyable. Her courage and passion for exploration were as boundless as the cosmos themselves. Together, they would shine light on this strange astrobiological puzzle. The old team, venturing forth into the unknown once again.

Chapter 3 - "The Quantum Jump"

Chapter3

0:00

/2:14

1×

The sleek corporate starship dropped out of faster-than-light travel, engines thrumming as it slowed to sublight speed. Through the oval viewport on the main deck, Kepler gazed in wonder at the spectacular sight before him - Cronus and its small red dwarf star. The planet was swathed in lush emerald and sapphire, with delicate wisps of cloud.

"Magnificent, isn't she?" Lilith murmured next to him. "I never get tired of that view after a quantum jump. Makes you feel so small though."

Kepler nodded. "I know what you mean. Sometimes I think we're tampering with forces beyond our comprehension."

Quantum travel enabled near-instantaneous journeys across interstellar distances by manipulating gravitational fields to warp space-time. But the process also caused Kepler to experience inexplicable dread. Humanity was using powers it did not fully understand.

"Attention passengers, this is your captain speaking." The intercom crackled to life. "We are on approach to Outpost Tempest on the southern continent. Please be seated and secured for atmospheric entry."

Kepler and Lilith took their seats as the ship pivoted gracefully. A high-pitched hum indicated the quantum shields being activated as they hit the outer atmosphere. Fiery plasma streamed past the viewports as the ship sliced through Cronus' skies.

Moments later, the turbulence eased, and Kepler caught a breathtaking glimpse of the planet's surface - alien forests interwoven with phosphorescent vines that seemed to glow and shift, defying earthly vegetation. Lily's fingers dug into his forearm excitedly. Then they were through the cloud layer, descending rapidly towards a cluster of sleek habitats nestled between lush purple trees.

With a few deft maneuvers, the ship came to rest on a landing pad. Kepler and Lilith donned lightweight suits and breathing gear calibrated to Cronus' atmosphere and climbed down to the planet's surface. A warm breeze rustled through alien saplings stirring their curiosity. What mysteries were waiting to be uncovered here?

Chapter 4 - "Arrival"

Chapter4

0:00

/3:18

1×

Kepler stepped off the landing ramp onto the soft loamy soil of Cronus. The purple trees surrounding the site swayed gently in the warm breeze. There was a sweet, floral scent on the air mixed with an earthy undertone. Above him, the sky was a dazzling azure with streaks of emerald green.

"Atmospheric composition check?" Lilith asked over their suit comms.

"Oxygen levels normal, nitrogen slightly elevated. Traces of unfamiliar organic compounds but breathable," Kepler reported, examining the readings on his wrist display.

After a moment's consideration, they retracted the faceplates on their suits, exposing themselves to the unfiltered alien air. It felt warm and heavy, laden with unfamiliar scents. Kepler detected floral notes like lavender and honeysuckle blended with deeper mosses. He felt his pulse quicken. After years traveling among the stars, this never got old - setting foot on a distant world for the first time. An unseen world, breathing its own air.

Lilith closed her eyes, embracing the new sensations carried on the gentle wind. "It's incredible..." she whispered. "I can almost feel it...life...”

They were interrupted by the approach of a stern-faced woman dressed in a Starfire uniform. "You must be the science team. I'm Commander Ada Wright, director of Outpost Tempest." Her manner was brisk and businesslike as she led them towards the sleek habitat structures nearby.

The interior of the base was spacious and softly lit, decorated with alien flora. Wright led them through the main atrium and labs. Researchers nodded in greeting, their eyes alight with purpose. This outpost was home to Cronus' best scientific minds.

"We've prepared quarters for you to freshen up. Then we can review the anomalies," Wright said as they entered the guest wing. Kepler and Lilith quickly stowed their gear and changed into fresh fatigues before joining Wright in the Situation Room.

Kepler studied the displays intently as Wright summarized the unexplained incidents - rapid growth patterns, bioluminescent reactions cascading through forests, even alleged movements by the vegetation. It was baffling, even impossible by Earth biology standards. What were they witnessing here?

Wright turned to them expectantly. "Well, any thoughts Dr. Arden?"

Kepler paused, meeting Lilith's eyes for a moment, then turned back to Wright. "This is unprecedented, Commander. We need to gather more data, then plan our next steps. But one thing is clear - there are forces at play here that we do not comprehend. Yet."

Lilith nodded resolutely beside him. Wright looked thoughtful. "Very well. We'll arrange your first excursion at dawn. In the meantime, our planetology archives are at your disposal."

As they left, Lilith leaned over excitedly. "This was a good call, Keph. I can feel it. Answers are close. We just need to listen to whatever Cronus is trying to tell us." Kepler hoped she was right. For all their sakes.

Chapter 5 - "Unearthly Greens"

Chapter5

0:00

/2:23

1×

The expedition rover glided swiftly through Cronus' verdant wilderness. Lilith sat in the driver's seat, expertly maneuvering around tangled roots and fallen trunks. Kepler rode shotgun, scanning their surroundings with various sensors and drones.

"We should be nearing the coordinates of the rapid growth event," Lilith said, checking her nav display. "I'm bringing us in closer."

Soon they arrived at a small clearing where the vegetation took on an altogether alien nature. The trees had an asymmetry to them, with spiraling trunks and branches dotted with shimmering leaves that seemed to ripple without any wind. The ground was carpeted by a spongy moss that gave off an eerie iridescence.

Kepler's scanners lit up with strange readings. "This is it. I'm detecting anomalous quantum signatures all across this biome." He collected several samples of the odd moss while Lilith deployed sensor drones to monitor the area.

"No obvious triggers, but quantum levels are spiking erratically," Lilith reported. "It's like the plants are...talking to each other."

Kepler stood up from where he was examining the twisting roots of a nearby tree. "You might be right, Li. Look at this." He showed her the root patterns on his holopad. The intricate meshes and whorls were slowly expanding and changing shape before their eyes.

Lilith's eyes widened in awe. "That's incredible! It's like the trees are reshaping themselves. But how?"

"My working hypothesis is that these lifeforms have evolved a symbiosis with some natural quantum field on Cronus," Kepler posited. "It lets them exchange energy and information rapidly to adapt their growth patterns."

"That would revolutionize astrobiology!" Lilith breathed. "Imagine if we could study and replicate it. But safely," she added warily.

Kepler nodded. "We have to proceed with great care and respect. This ecosystem is far more complex than we realized."

As if in response, a cascade of bioluminescent pulses ran across the mossy ground, mirrored by the shimmering leaves above. Then all was still again.

Lilith met Kepler's eyes. "Did you see that? It was like a reply..." Kepler stood transfixed. What had they stirred awake?

Chapter 6 - "Quantum Bloom"

Chapter6

0:00

/2:19

1×

Kepler stepped cautiously through the alien biomes, scanning a flowering vine that wound its way around a tree trunk. Its petals rippled with colors that had no name in any earthly tongue. Hints of indigo, violet, and crimson undulated across their surfaces like celestial clouds.

"This region has the highest quantum levels so far," he said over his shoulder to Lilith. She was engrossed in a nearby plant, whose woody stems twisted in impossible shapes.

"Their cell structures have evolved to use ambient quantum fluctuations here as an energy source," Lilith marveled. "The mineral composition in the plants is acting like a photosynthetic enhancement. It's how they get so large and complex."

She touched a finger to one of the violet petals, which immediately flashed a bright turquoise across its surface. The reaction cascaded through the rest of the vine, racing down to the roots.

Kepler's eyes widened. "Did you see that? It's like the lifeforms are all connected here."

Lilith nodded excitedly. "It must be a quantum communications system! The mineral elements allow energy transfer and information sharing between the species."

As if to confirm the theory, a shimmering pulse began to spread slowly through the surrounding vegetation. It built among the leaves like an aurora, then moved in waves along the ground, patterns flickering across the mosses.

A moment later, it reached Kepler and Lilith, washing over their legs in a tingling energy flux. The eerie beauty took their breath away. They seemed to detect rhythms and motifs within the dance of light, reminiscent of whale songor murmuring voices.

"Can it sense us?" Lilith whispered in awe. In answer, the aurora built to a crescendo around them, then faded slowly away. A deep silence followed, broken only by the wind in alien boughs.

Kepler met Lilith's wondering gaze with a mix of euphoria and trepidation. "This life," he said slowly, "It's connected at a level we've never seen. A living quantum network."

What they had stumbled upon was changing everything Kepler knew about the boundaries of life.

Chapter 7 - "Echoes of Sentience"

Chapter7

0:00

/2:11

1×

Kepler sat in the darkened monitor room of the Cronus research base, surrounded by displays tracking the planet's exotic quantum biome. He rubbed his eyes wearily after another near-sleepless night spent analyzing the alien ecology's perplexing rhythms and cascades.

The base's doors slid open and Lilith entered, clutching two mugs of steaming coffee. "No luck figuring it out?" she asked sympathetically, handing Kepler a mug as she peered over his shoulder at the hypnotic readings.

Kepler sighed and took a grateful sip of the coffee. "Their quantum communication network exceeds anything I've ever studied. The patterns nudge at my brain like half-heard whispers, but the meaning eludes me."

Lilith pulled up a chair, studying the monitors as she sipped her own drink. "I know what you mean. It reminds me of whale song - familiar cues and motifs but in a foreign language."

They sat in pensive silence until a new flurry of patterns materialized on the screens - rippling colors coalescing into an ethereal shape reminiscent of the luminescent Cronian foliage.

Lilith's eyes widened and she put her mug down slowly, transfixed by the display. "Is that...one of their lifeforms?"

Kepler could only nod mutely, equally enthralled by the delicate phantom flora suspended amidst the data readouts. He leaned forward, struggling to discern some meaning in its flowing geometry as it gradually dissolved into quantum static.

"A representation of their world..." he murmured in awe. "Do you think it could signify sentience?"

Lilith met his gaze, her expression mirroring Kepler's mix of fear and fascination. "If so, it's nothing like any intelligence we've ever encountered. The Cronians seem intrinsically linked to the quantum forces permeating this entire planet."

Kepler had no chance to reply before proximity alarms blared urgently through the base. He exchanged an anxious look with Lilith and they rushed to the doorway where Commander Wright stood, staring outside with disbelief.

Chapter 8 - "Across the Stars"

Chapter8

0:00

/3:55

1×

Kepler shielded his eyes against the intense blue-green glow emanating from beyond the habitat structures. Strange wispy shapes swirled hypnotically in the sky above the forest canopy.

"What is that?" Lilith gasped, as awestruck researchers gathered around them.

"Some kind of atmospheric energy manifestation," Kepler said slowly. "But I've never seen anything like it."

The glowing apparitions continued their otherworldly dance. Almost like aurora borealis, but somehow more solid, more purposeful. As Kepler watched, eyes wide, he could have sworn the flowing forms kept morphing subtly into different alien shapes.

"It's the Cronians," he whispered. "Trying to communicate with us."

Lilith shot him a startled look. "Are you sure?"

He pointed up at a shape that now resembled a large ethereal flower - similar to what they had glimpsed on the monitors. "It's similar to the patterns we've been recording from the planetary quantum network. This must be a manifestation of the native intelligence."

The apparitions had coalesced into seven shimmering pillars, spaced evenly above the habitat. Kepler's mind raced. What was the significance of the number seven? A code? Some fundamental mathematical constant for the aliens?

As he watched, the pillars began emitting strange harmonics almost below the range of human hearing. Kepler's bones seemed to resonate with the tones. Lilith put a hand to her chest in alarm.

"Can you hear that?" Kepler asked breathlessly. The researchers around him wore expressions of apprehension and awe.

Lilith nodded mutely, her face bathed in rippling teal light.

The resonating pillars were unlike anything Kepler had ever encountered. There was structure, perhaps even information, encoded in this display. Intelligence - alien, unknowable, but intelligence nonetheless.

After several hypnotic minutes, the glowing apparitions simply faded away, dissolving like mist above the shadowed forest. An awed hush lingered.

Lilith turned to Kepler, her eyes reflecting the residual light. "You said it yourself, Keph. We're tampering with forces we don't comprehend. I think it's time we contacted Earth."

Kepler nodded grimly. As much as it pained him to admit, they were out of their depth. These lifeforms were communicating through channels humans couldn't even perceive. What consequences were they stirring into motion?

Back in the comm center, Kepler composed an encrypted data packet for transmission back to Earth via quantum relay. Video clips of the hypnotic apparitions, scans of the bizarre alien flora, patterns of light rippling across the plant life - all the bewildering data they had gathered so far.

He included a personal log, detailing his concerns about protecting the Cronian biosphere, and calling for ethical oversight from Earth authorities. Were Voronin and his corporate backers prepared for the implications of this discovery? Kepler had a duty to raise his misgivings, before the situation spiraled beyond control.

As Kepler queued up the packet and hit 'Send', he felt the weight of the unknown settling around his shoulders. What forces had they awakened under this alien sun? The vast gulf between human and Cronian minds left him adrift in an ocean of questions with few reference points to guide his moral compass.

Beside him, Lilith slipped a gloved hand into his and gave it a comforting squeeze. Together, as always, they would find a way. The transmission signal pulsed one last time - carrying humanity's first faltering reach across the stars, to make contact with an utterly alien kind.

Chapter 9 - "Cronian Council"

Chapter9

0:00

/5:07

1×

The shadowed Cronian forest took on a foreboding aura as Kepler and Lilith made their way to the coordinates specified in the alien message. It had arrived that morning - a rhythm of lights flickering through the exotic plants, coalescing into shapes reminiscent of the glowing apparitions in the sky.

The message was cryptic yet compelling. After hours spent analyzing the pulsating patterns, Kepler determined the Cronians were inviting them to some kind of meeting. But the specifics eluded human comprehension.

Lilith was apprehensive. "We're venturing far from base. And we have no idea what we'll actually find out there." She checked her gear - backup oxygen, emergency beacon, sidearm. Precautions offered some reassurance against the unknowable.

"It's a risk we have to take," Kepler said. "This could be a pivotal moment of contact. We can't hide in our outpost forever." Lilith nodded reluctantly.

As they entered a wide clearing, Kepler signaled a halt. The vegetation here had assumed an altogether alien nature - woody trunks spiraling up in fractal contortions, silvery leaves that chimed wildly without wind. No familiar earth references remained, like wandering into the madness between galaxies.

"How will we even recognize the Cronians?" Lilith whispered nervously. As if in response, a haunting harmony arose slowly from the forest, reverberating through Kepler's bones. Shapes began to coalesce in the clearing, fashioned from shimmering prismatic mist.

Seven figures stood before them now - at least, Kepler assumed they were figures. There was a disorienting fluidity to their outlines, colors and patterns swirling hypnotically within. He could discern no faces, limbs, any features relatable to organic life. Yet their presence pressed in like a heavy weight.

"The...Council, I presume?" Kepler ventured. The harmonic tones shifted in reply. Kepler pointed to himself and Lilith.

"Human. We wish to...understand." The resonances took on a more rapid modulation, as if in animated debate. Lilith looked anxiously at Kepler. One of the Cronians glided closer, and Kepler fought the instinct to flee.

A vivid aura unfolded around the entity - somehow separate from its core, while still a part of it. Colors and shapes built swiftly, until Kepler gasped and stepped back. Before him was a miniature diorama of Outpost Tempest and the surrounding biodome rendered in minute faithful detail. It was like looking at a model reflection in midair.

"A gift..." Kepler breathed in awe. The being had read his very thoughts and reproduced the human settlement with startling accuracy. He bowed deeply in thanks. The resonant discussion rose in a new timbre - perhaps indicating approval. Or at least, Kepler hoped it did.

Another Cronian drifted forward, aura shifting to depict Cronus' bizarre biomes and the rippling quantum dance across the exotic lifeforms. Kepler's mind reeled at the impossible complexity encoded in every swirling particle.

Next, Orion Voronin's scowling face materialized briefly before the aura dissolved in discordant tones. Kepler understood this stark message - the Cronians knew of Earth's corporate exploitation. And were displeased.

"We come in peace," he implored. "We wish to learn." The resonant harmonics conveyed skepticism. Lilith pointed urgently skyward. The shimmering apparitions had reappeared, their numbers greater now. A visible reminder of the Cronians unfathomable powers.

Kepler spread his hands in a gesture of appeasement. "Help us understand you, so that we may prevent misunderstanding." The resonances became contemplative, then inquisitive.

One Cronian replicated Kepler's gesture precisely with an aura-arm. Another flashed mathematical symbols of increasing complexity that Kepler struggled to decipher. Were they testing the limits of human comprehension? He focused intently, intuiting patterns that hinted at meaning.

Hours passed in hypnotic mental engagement. The gulf between human and Cronian minds strained but did not break. Slowly, painfully, the first fragile bridges of understanding were being forged.

As dusk fell, the Council vanished in a swirl of prismatic mist. The resonances withdrew like a receding tide, leaving Kepler and Lilith spent but also strangely exalted. They had glimpsed the unimaginable complexity that lay across the galactic divide. And for a few precious moments, sentience had reached out to sentience, exchanging gifts of mutual comprehension.

The long night back was filled with luminous dreams. Dreams of resonance, rippling and binding the stars themselves.

Chapter 10 - "Orion's Ambition"

Chapter10

0:00

/3:29

1×

Earth shone like a blue marble in the void as the sleek Starfire shuttle approached. Inside, Orion Voronin gazed pensively at his home planet, steeling himself for the coming storm.

Kepler's report from Cronus had ignited a firestorm back at corporate headquarters. Requests came flooding in to declare the planet off-limits, to treat the alien lifeforms with reverence. Orion scoffed inwardly. Fools, all of them. They didn't understand what was at stake.

He straightened his collar as the shuttle entered low orbit, bound for Starfire's towering HQ. His family's legacy hung around his neck like a millstone. Three generations of Voronins had built this company from nothing, exerting domination over first Russia, then Earth and beyond. And it fell to Orion to push the frontier ever further.

As he strode into the plush executive boardroom, all eyes turned to him. The board members wore guarded expressions, wielding Kepler's revelations like a weapon against him. Orion raised his hands diplomatically, hoping to disarm them.

"My friends, I understand your concerns. But think, have we not faced situations like this before as explorers? The temptation to flee from the unknown?" Murmurs rippled around the table. Orion pressed on.

"What if those first sailors had turned back in fear of edge of the world instead of discovering new continents? Where would we be now?" He infused his voice with passion, playing on their human desire for knowledge and conquest. It was a risk, but Orion relied on his rhetorical skills to bring them around.

"This discovery on Cronus heralds a new age of exploration for Starfire and humanity itself. Yes, there are risks and unknown factors. But think of the potential rewards! Are we not called to rise to the challenge, as our forebears did?"

He could see them wavering, seduced by his words despite their misgivings. Orion moved in for the kill. "We cannot allow irrational fear to rob us of knowledge and technological advancement. I would not be here today if my ancestors had balked at the bold course."

Nods and murmurs of assent rippled around the table now. Orion allowed himself a small smile of satisfaction. He still had the common touch.

"Therefore, I propose we increase operations on Cronus immediately. Establish outposts across the planet to tap its unique quantum resources. With prudent safeguards, of course." This concession brought more affirmations from the group.

The board chairman turned to Orion with grudging respect. "Your arguments are compelling, Voronin. Very well, we will proceed as you recommend."

Orion gave a deferent bow of his head, while inwardly exulting at this victory. He would show Kepler and the other doubters. Harnessing Cronus' quantum forces was merely the first step. The very cosmos would open to him, Orion Voronin, like a lover swooning at his feet. He was so close now. No one would deny him his rightful place among the gods.

As his shuttle descended back to Earth, Orion gazed out at the glittering Starfire headquarters dominating the skyline, imagIning it eclipsed by even grander monuments to himself someday. All of humanity's knowledge and power would be his. But first, he had an alien planet to conquer.

Chapter 11 - "Kepler's Moral Dilemma"

Chapter11

0:00

/3:43

1×

Kepler sat alone in his spartan quarters, head in hands, struggling to order his turbulent thoughts. Ever since their first true contact with the Cronians, a nameless dread had taken hold in his heart, gnawing at him during every waking hour. And even seeping into his uneasy dreams.

He understood so little - about the alien intelligence they had encountered, the quantum forces enabling the planet's exotic biosphere, or the consequences of humanity's continued meddling. But that was exactly the root of his anguish - comprehending just enough to grasp the perils of their ignorance.

With a heavy sigh, Kepler stood up and paced the small room, his mind returning to that illuminated clearing in the forest, the haunting resonances that chilled his very bones, the hypnotic auras...and the unmistakable warning against Orion Voronin's ruthless exploitation. Exploitation that even now was accelerating despite Kepler's strident objections.

His scientific career had been dedicated to understanding life's boundless diversity across the cosmos. But this was the first time Kepler felt the profound weight of responsibility - not just to study, but to shield and nurture. The Cronians' sentience was intricately tied to the quantum forces permeating their world. Forces that his own species blithely sought to harvest, without pausing to consider the consequences.

Outside the curved window, Kepler gazed at the lush Cronian night, where even the darkness seemed somehow aglow. This world was ancient beyond knowing, and now it lay at the mercy of short-sighted corporate interests represented by men like Voronin. Kepler rested his head against the cool pane, feeling utterly powerless.

As an explorer, he had always clung to the hope that expanding knowledge could uplift his fellow humans, make them act more nobly. But now, that hope wavered in the face of Voronin's monumental arrogance and ambition. What if enlightenment never came for humanity? What if mankind's ignorance doomed them to perpetual destruction?

A light knock stirred Kepler from his bleak reverie. Lilith entered quietly, her presence radiating compassion. His dear friend always sensed when darkness descended upon his thoughts. Without a word she embraced him, anchoring Kepler amidst the raging storms.

"Talk to me, Keph," she entreated softly. "Let me help shoulder this burden."

Haltingly, Kepler shared his inner turmoil, giving voice to all his doubts and fears. About humanity's path, their place in the universe, his own conflicted purpose. Lilith listened silently, letting the tide of anguish flow, until at last Kepler lapsed into an exhausted silence.

Lilith took his hands then, her eyes luminous but steady. "Your doubts reflect the goodness in you, Keph. Keep nurturing that seed, and let it blossom into wise action."

Kepler swallowed hard, moved by her faith in him. She was right - wallowing in despair served nothing. He could yet do some good, if he kept his moral compass aligned.

With renewed vigor, Kepler vowed to confront Orion's relentless exploitation and safeguard the Cronian ecology. Perhaps enlightenment would come to their species, perhaps not. But he must live and act as though the dawn might break at any moment. With Lilith's kindness lighting the way, Kepler felt the shadows over his heart lifting. A glimmer of hope amidst the dark cosmos.

Chapter 12 - "Orion's Dismissal"

Chapter12

0:00

/5:48

1×

The sleek corporate shuttle descended through Cronus' hazy orange skies towards the gleaming cityscape of Voronin Point, headquarters for Starfire's interstellar operations. Kepler gazed broodingly out the window as the alien metropolis came into focus below, its silver towers rising incongruously from the primordial landscape.

He had argued vehemently against further colonization, but Orion Voronin would hear none of it. His ambitions were fixed upon harnessing the planet's quantum forces for his own glory, indifferent to the consequences for Cronus' fragile ecology. Or the unimaginable sentience they had encountered in the forest.

Now Kepler was summoned before the CEO to give a "progress report" on his team's research. But he knew Orion's mind was already made up. The man's arrogance and hunger for power seemed boundless. How could Kepler make him see reason?

The shuttle glided between towering corporate habitation modules to land smoothly at the entrance of the Executive Spire, Starfire's seat of power on Cronus. Kepler steeled himself as he entered the soaring atrium and presented his ID chip to the armed guards. One scanned his biometrics and nodded curtly.

"Mr Voronin is expecting you. Top floor."

Kepler stepped into an express lift that whisked him up at dizzying speed. Before he could even collect his thoughts, the doors opened directly into Voronin's palatial office. Floor-to-ceiling windows presented an awe-inspiring panoramic view of the city below and the untamed Cronian wilderness beyond.

But Kepler's gaze was drawn to the lone figure seated behind a massive teak desk, framed against the vista like some feudal lord. Orion Voronin slowly raised his head to fix Kepler with an intense stare.

"Kepler, welcome back." His tone was cordial, but his cybernetically enhanced eyes were cold and appraising. "Please, take a seat."

Kepler sat hesitantly before the imposing desk. He took a deep breath, choosing his next words carefully.

"Orion, we have to talk about Cronus. The quantum forces here are beyond anything we've dealt with before. We risk devastating consequences if we continue to exploit this world recklessly."

Voronin templed his fingers, regarding Kepler impassively. "And what evidence supports these concerns?"

Kepler leaned forward earnestly. "The very nature of the planet's ecology for one, intrinsically linked to ambient quantum fields. And the Cronians themselves..."

He described their eerie first contact with the aliens in the forest, the hypnotic resonances, the incomprehensible complexity and sentience.

But Voronin's expression remained unmoved. When Kepler finished, he simply gave an indifferent shrug.

"A quaint account. But I expected concrete data from our Chief Astrobiologist, not fairy tales." His tone turned mocking. "For all you know, these 'apparitions' are some natural planetary phenomenon. An overactive imagination does not justify halting our operations."

Kepler struggled to remain calm. "With respect Orion, there is too much at stake here to dismiss the risks. Your actions could irrevocably damage a complex alien biosphere. Does that not give you pause?"

Voronin rose abruptly from his seat and turned to the expansive window, hands clasped behind his back arrogantly.

"I didn't bring you all the way to Cronus to wax philosophical, Kepler. I need options to harness this world, not excuses."  He kept his back to Kepler. "Unless you can provide me with definitive solutions soon, I will find researchers who can."

Kepler stood slowly, trying to contain his swirling outrage, frustration and disappointment.

"I cannot in good conscience offer you technical advice for exploiting this planet and its inhabitants. I'm sorry, Orion."

Voronin did not turn around.

"Then we have nothing further to discuss. You are dismissed, Dr Arden." His tone was icy.

Kepler exited the office in weary silence. The towering atrium with its gleaming surfaces and scurrying corporate workers now seemed garish and empty to him. He felt only a deep sense of sorrow. Voronin's hubris had closed his mind, and Kepler lacked the power or persuasiveness to open it again.

But he would not abandon Cronus. Even if he could not sway Orion, perhaps the Cronians themselves would find a way to halt the human desecration of their world. Kepler clung fast to that fragile hope. Orion was but one man in the end, however arrogant - but the will of an entire sentient species was another matter.

Kepler stepped out of the massive Executive Spire onto a landing platform overlooking the sprawling Voronin Point complex. The wind carried a faint harmonic resonance that only Kepler could detect. As if sensing his thoughts, the Cronians were signaling  their patience was finite against the human onslaught. Subtle power rippled across this ancient land, far older and stranger than the petty earthly authority Orion Voronin brandished so brazenly.

Kepler raised his eyes to the swirling violet skies, stained crimson by the setting of Cronus' distant sun. Between Voronin's ruthless ambition and the Cronians' cryptic power, the fate of both species now balanced on the edge of a knife. All Kepler could do was follow his conscience as events unfolded. Eyes open, however bitter the path ahead. With a heavy heart, he boarded the shuttle to begin the long journey back.

Chapter 13 - "Indigenous Defense"

Chapter13

0:00

/4:00

1×

A baleful crimson dawn crept over the horizon as Kepler stepped out of the remote research outpost onto Cronus' barren plains. The morning sky was stained a virulent purple, flickering with ominous auroral shapes. Kepler shuddered, unnerved by the planet's changed temperament.

For weeks now, tensions had been rising as Orion Voronin's ruthless exploitation widened across Cronus. Desperate warnings from Kepler went unheeded, and attempts to reach out to the Cronians were met with stony silence. The planet seethed with growing unrest.

Now Kepler led a small team dispatched to investigate garbled reports of technology failures at this isolated outpost. As they had flown in under the angry sky, Kepler sensed they were already too late.

The base's autonomous systems were offline, doors sealed shut and communications dead. After donning protective suits, Kepler managed to override a stubborn airlock using the Outpost Director's authority code. As the inner door cycled open, a disturbing odor wafted out - burnt metal and ozone. Kepler exchanged an apprehensive look with Lilith before stepping inside.

Charred cables snaked across shattered floor panels in the gloomy corridors. Not a system or screen remained intact. Flickering emergency lights revealed an ominous trail of corrosion eating through the base infrastructure. Kepler's skin crawled as he traced the unnatural decay back towards the main laboratory.

The sprawling chamber was utterly devastated. Kepler picked his way silently through the wreckage - twisted remnants of robotics and electronics, databanks mottled by creeping decay. At the room's center, the base's Cronium reactor stood lifeless. The shimmering mineral - so vital to Voronin's ambitions - had blackened like necrotic flesh.

Lilith activated a malfunctioning monitor, wincing at the fizzing display. "It's like everything just...dissolved from quantum effects. But how?"

Kepler turned slowly, realization dawning. "This was no accident or system failure. The Cronians did this - a targeted retaliation."

Lilith stared at him. "You mean this was an attack? But why only shut down instead of destroying the base?"

"A warning," Kepler murmured. "Demonstrating their power to incapacitate our facilities with pinpoint manipulation of quantum fields. The blighted reactor is a message - Cronium turned poisonous at its source."

As if in response, a resonant Croanian harmony echoed through the ruins, more dissonant than before. Kepler's heart froze at the palpable anger infusing the alien tones. This was only the first act.

Around Cronus' settlement, panic erupted as transportation, communication and defense systems mysteriously failed. Voronin raged helplessly against the planetwide breakdown. Deep in the forest, the Cronians' shimmering pillars intensified as the barriers between the quantum and physical worlds thinned dangerously.

In Kepler's mind, a single word echoed with crystalline clarity - "Depart." The ultimatum was clear. Abandon Cronus or face complete dissolution, not just of technology but reality itself. Mankind had tried the Cronians' patience for too long.

Kepler's hands trembled as he placed a scrambled call to Voronin's inner council, warning of imminent catastrophe. Even now Orion plotted retaliation, unwilling to relinquish his transgalactic ambition. But Kepler knew this was no longer a human conflict. What power did earthly authority have over beings who spoke with the very forces of nature? With stakes this high, he and Lilith might be the only conciliatory voices left between two irreconcilable species hurtling towards mutual annihilation.

Chapter 14 - "Cosmic Crisis Averted"

Chapter14

0:00

/4:12

1×

Kepler raced through the quantum research complex, past teams of scientists frantically trying to contain the cascading Cronian retaliations. Alarms blared as precious equipment flickered and dissolved under the unearthly assault.

Lilith stood at the heart of the control room, desperately calibrating protective field modulators. But each quantum-laced wave from the Cronians slipped through, leaving behind creeping decay.

"I can't stabilize any of the shield grids," Lilith yelled over the din. "It's like their powers are permeating reality itself!"

Kepler gripped her shoulder. "We have to keep trying. If we can just shield Voronin Point, it may pacify them." But inwardly, he harbored little hope. The Cronians' vengeance was an unstoppable tide.

Suddenly, the shrieking alarms fell eerily silent. Across the complex, screens winked out and reactors went dark as oily corruption swallowed them. A message chimed through Kepler's COMM implant in his skull - evacuation protocol activated. The entire staff of Voronin Point had minutes to flee into orbit before the Cronians unraveled the fabric of Spacetime around it.

Kepler and Lilith stood frozen amidst the dying technology, bathed in the glow of emergency lights. After a lifetime exploring the cosmos, to die here under an alien sun... it seemed so senseless.

Lilith slipped her gloved hand into Kepler's. "It's been an honor, Keph." Her voice trembled butKepler sensed the deep well of courage within her.

As the evacuation klaxon wailed, Kepler embraced her, wishing they had

more time. His thoughts turned to sunny days on gentle Earth, so far away now. No regrets shadowed those memories, only gratitude for the wonders they had witnessed.

"I love you, Lily Carson," Kepler whispered. She smiled bravely.

"And I love you, Kepler Arden. Don't be afraid."

As reality collapsed around Voronin Point in a contagion of entropy, Kepler clung to the reliable anchor of Lilith's spirit. Eyes closed, they waited for the end.

But it did not come. The seconds stretched on, and the satisfying oblivion refused to claim them. Kepler's eyes flew open in shock. The creeping decay inches from them had halted, frozen in place like varicose veins across spacetime.

Lilith gasped, her fingers tightening painfully on Kepler's. "Are we...alive?" she managed in an awed whisper.

Kepler shook his head in disbelief. "For now...but how?"

Around them, the staff emerged from hiding places, equally stunned to still exist. Amidst the chaos, Kepler's comm implant crackled.

"Kepler, can you hear me? You're alive!" The voice of Sal, Cronus' ambassador to the Cronians, rang through. "The Council is holding back the planet's fury, but can't restrain it much longer!"

New strength surged through Kepler's heart. By some impossible grace, they had a second chance. "Sal! We need to reconvene emergency talks. There must be a way to make amends!"

Sal's response held a thread of hope. "Come to the crystal grove, quickly! The Council awaits."

As Kepler and Lilith scrambled for a functioning rover, Cronus groaned ominously around them. The surface rippled as spacetime strained against its quantum chains.

They sped under the dark sky towards the towering crystal grove, last stand between two species. Inside Kepler's mind, a fragile trust took root - that maybe peace was still possible, through compassion and sacrifice from both sides.

He prayed silently it was not too late. The long-term threat still loomed over Earth and Cronus alike, as pressure built against the precarious walls holding back mutual destruction. But with wisdom and patience, the worst fate could yet be averted.

Kepler and Lilith clung fiercely to that faith as they plunged into the luminous Cronian forest, towards a new hope.

Chapter 15 - "Cronians' Protest"

Chapter15

0:00

/5:25

1×

A deathly silence hung over the crystalline grove as Kepler and Lilith approached. The alien trees towered overhead, refracting the wan sunlight into splintered rainbows across the mossy floor. There was no sign of Sal or the Cronian Council.

Unease gnawed at Kepler's gut. The planet's very substance felt on edge, shivering with barely contained forces. He opened his mouth to call out, but an urgent gesture from Lilith stilled him. She pointed around the grove, eyes wide.

Kepler focused on their surroundings, and his breath caught in his throat. The Cronians were already here - not in any anthropic form, but diffused through the quantum rhythms of Cronus itself. Strange harmonics pulsed across trunks, leaves, and soil, welling up from the planet's depths. Kepler's bones resonated alarmingly with the alien tones.

"Where's Sal?" Lilith whispered. Kepler shook his head helplessly. They were alone - perhaps the last humans left on Cronus as the evacuation continued above. What fate awaited them amidst these cycloean trees?

As they stood frozen, the harmonic pulsations rose menacingly. Kepler cried out as an arctic beam lanced through his skull. His knees buckled under the psychic assault. Through fading vision, he saw Lilith collapsed nearby, blood trickling from her nose.

"What...do you want with us?" Kepler managed to rasp. There was no reply but the mounting alien crescendo. Kepler sensed fierce debate amongst the entities - to crush these two insolent humans, or spare them as witnesses?

Just as oblivion loomed, the attack ceased. The grove grew deathly still, the very sunlight muted and gray. Kepler crawled agonizingly over to Lilith's crumpled form, cradling her head. Her eyes fluttered open, glazed with pain.

"We're still breathing..." she croaked. They held each other close amidst the silence, taking comfort in the miracle of their fragile existence. But the reprieve was temporary. Cronus seethed around them, channeling the Cronians' wrath.

Kepler's communicator suddenly crackled to life, Sal's panicked voice filtering through. "...stopped all quantum travel near Cronus...some kind of interstellar block..." The signal dissolved into hissing static.

Cold understanding gripped Kepler. The Cronians were no longer confining their ire to Cronus alone. Their distress signal was resonating across the very fabric of space-time, freezing all quantum-enabled travel and communications. Supply runs, colonist transports, all interstellar activity - Earth's lifelines were being severed.

A fierce debate erupted on the Cronian Council, rippling through subspace itself. What consequences were too severe for the invaders defiling their world? Kepler's heart hammered against his ribs. Would they isolate mankind completely, leaving Earth to wither and die?

"Please, have mercy..." Kepler beamed his thoughts desperately towards the shimmering trees, swaying under the weight of this decision. "We seek reconciliation..."

The psychic assault returned - not to crush, but to envelop Kepler and Lilith completely. Their human senses were engulfed by the Cronian consciousnesses, minds linked across lightyears of radical otherness. For a few eternal moments, they saw as the Cronians saw.

Civilizations flickered by faster than light, rising and falling like mayflies. Yet each precious, worthy of guardianship. The cosmos was a sacred trust. How to correct without destroying? Kepler glimpsed the Cronians' ferocious anguish at humanity's trespass, but also their reluctance to become executioners over one young, foolhardy race.

Just as quickly, Kepler and Lilith were released, minds raw and ringing. Lilith was curled on the spongy moss, weeping softly. "Such pain..." she murmured. "Their world, their kindred..."

Kepler helped her stand gently. "There is still hope." He felt the Council's hesitation, their unwillingness to doom mankind entirely.

A single resonant tone reverberated through the grove, at once devastating yet restrained. The Cronians had delivered their ultimatum across all of Earth's interstellar colonies and outposts. For now they stayed their hand from further retaliation, awaiting humanity's response.

But time was short. Word would soon reach Voronin of this crisis hampering his transgalactic ambitions. How would the CEO react? Kepler dreaded that confrontation almost as much as the wrath of the Cronians. With two implacable forces competing to determine mankind's fate, it seemed oblivion was still the inevitable outcome.

But Kepler clung fiercely to hope as they made their way back through the darkling woods. Past mistakes could not be undone, but with wisdom and compassion, destruction could still be averted. The futures of both species now hung by a thread. The duty fell to Kepler to guide humanity back from the brink, or become architects of each other's demise.

Chapter 16 - "A Race Against Time"

Chapter16

0:00

/2:57

1×

Kepler gripped the rover's controls, deftly swerving around tangled Cronian roots as they raced back to Voronin Point. Beside him, Lilith clung to her seat, eyes shadowed with the weight of the knowledge they carried.

The Cronians' psychic message had laid bare the peril facing both species - oblivion for one, survival for none. Kepler's mind churned with desperate questions. How to make Orion see reason and abandon his reckless colonization? And how to appease the Cronians before their retaliation severed Earth's interstellar lifelines completely?

Time was bleeding away faster than lightyears under quantum drive. Even now, Orion roused his mighty corporate armada, spurred by rage and arrogance. Kepler knew that fleet would set course for Cronus within hours to crush this upstart indigenous force defying the CEO's ambitions.

Yet if Orion's warships reached Cronus, humankind would forfeit any last shred of trust or mercy from the Cronians. Kepler harbored no illusions - a technology-driven onslaught would be swatted aside like gnats by beings who spoke with the very forces Orion sought to enslave.

As the Voronin Point towers came into view through the swirling violet mists, Kepler felt the heavy gaze of the Cronians upon them. The entities wavered between hope and fury - though their ultimatum had spared mankind outright dissolution for now, restraint wore thin. Even Lilith's courage wavered under that pitiless regard from the clouds.

Yet in Kepler's heart, determination blazed brighter. Together with Lilith, they would stand before the demigod Orion and awaken some glimmer of wisdom in him, or die in the attempt. And if Orion proved intractable, they still had one slim chance - to broker a peace directly with the Cronian Council.

Kepler knew well the risk they invited, laying their fragile human minds open again to that unfathomable psychic torrent. But no other path remained to stop Orbital bombardment and learn the lessons Cronus was meant to teach.

The rover skidded to a precarious stop outside the Executive Spire as alarms blared within. Kepler took Lilith's hand, steeling himself and drinking in her beloved face one last time. Then, they strode into the tower to try and turn the tide of history.

One path led to salvation, the other to extinction. But the choice still hung in the balance. Kepler clung to that tenuous hope as they entered the great predator's lair. Guiding humanity through this crisis was their sole purpose now.

Human and Cronian fates would be forged here today, beneath an alien sky. Kepler silently prayed their wisdom and sacrifice would suffice to redeem both races from the yawning abyss.

Chapter 17 - "Orion's Relentlessness"

Chapter17

0:00

/4:08

1×

The door to the Executive Spire's situational chamber slid open and Kepler froze, momentarily overwhelmed by the sheer controlled chaos within. Teams of corporate officers and advisers swarmed around displays tracking Earth's crumbling interstellar infrastructure. At the center, Orion Voronin stood like the eye of a storm, barking orders and demands for solutions.

Kepler and Lilith hesitantly approached the CEO, whose wild gestures and ranting drowned out all else.

"—the entire Cronus operation in shambles! Quantum travel disrupted across colonies!" Orion slammed a fist down. "Unacceptable! I want our astrophysicists and engineers on this disaster immediately!"

"Orion!" Kepler had to shout to be heard. "This wasn't an accidental systems failure. The Cronians themselves are blocking all quantum travel and trade. Your plans have provoked them across the entire sector!"

Orion whirled, his face mottled with rage. "Provoke them? Those apparitions you ranted about?" He laughed harshly. "I don't care if they're ghosts or gods, they will learn to bend knee when I am through with them!"

Kepler struggled to hold back despair at Orion's arrogance. "Please, you must listen! The Cronians wield power beyond our comprehension through quantum manipulation. Our technology is nothing to them."

But Orion seemed beyond reason, consumed by indignation. "You always were too timid, Kepler. These pathetic aliens don't realize who they're dealing with!" He turned to his advisors with palms raised theatrically.

"We shall respond with the full might of the Voronin family's Star Empire! Ready the Cronus retaliation fleet for immediate departure!"

The chamber descended into frenzied activity as officers rushed to carry out his command. Kepler grabbed Orion's arm in supplication, heedless of the guards that stepped forward threateningly.

"No! You'll only ensure humanity's destruction if you confront them with force. Please Orion, even you can't stand against beings who speak the language of reality itself!"

For a heartbeat, Lilith glimpsed something through the CEO's rage - a profound exhaustion shadowing his eyes. The burden of his family's lofty legacy weighing down his shoulders. She pressed Kepler's arm warningly as Orion gently withdrew himself.

"Your concerns are noted, Dr Arden," Orion said quietly. "But I must uphold the Voronin name. At any cost." He turned away, shoulders slumped. "Guard, please see our guests out."

Kepler struggled uselessly as they were led firmly to the doors. There, he turned back once more.

"This madness won't bring you peace, Orion! Billions will perish!"

The closing doors muted his cries. Lilith hugged Kepler as despair threatened to overwhelm him. Voronin's greed for power and glory had sealed both their fates.

Yet amidst anguish, Lilith sensed Orion was not an evil man, but one crushed under ambition's merciless weight. She whispered close to Kepler's ear.

"All is not lost yet, my love. We must make directly for the Crystal Grove."

Kepler turned wild eyes upon her. "And say what? Plead for our species' survival?"

She smiled sadly. "If we can first show the Council humanity, perhaps they will show us mercy in kind."

Kepler shuddered with fresh hope and fear. As Orion's armada streaked Cronus-bound, only one slim chance for reconciliation remained - to open their minds and souls to the Cronians without restraint. And pray it would be enough to halt extinction's relentless hand.

They clung to each other under the shrouded sky. If these were to be humankind's final hours, they would face the end together, with hearts open. Lilith's courage kindled Kepler's spirit anew for the trial ahead. Hand in hand, they made for the crystalline forest to try and ransom humanity's very soul.

Chapter 18 - "Cronian Ultimatum"

Chapter18

0:00

/4:13

1×

Kepler and Lilith entered the crystalline grove as Cronus' sun sank below the horizon. The alien trees surrounding the clearing thrummed with an ominous resonance just beyond human hearing. Kepler felt it vibrating through his bones - a harmonic frequency conveying urgency...and barely restrained fury.

"They know," Lilith whispered, wide-eyed. Kepler could only nod grimly. Orion's retaliation fleet was streaking towards Cronus even now, bristling with enough firepower to shatter moons. Yet those technologically mighty armaments would be less than toys against beings who effortlessly unraveled quantum skeins with a thought.

Kepler opened his mind, beaming out a plea into the charged ether. "Hear us! We come in peace, seeking reconciliation!"

The thrumming tone modulated into rapid oscillations, akin to heated debate. Lilith stepped closer to Kepler as the lambent crystal trees shivered. A fierce war of wills raged around them - the Cronian Council weighing mankind's fate.

Finally, the trees stilled. A single pure note reverberated through the grove poignantly. Then, as one, the entities materialized - seven shimmering pillars, featureless yet undeniably sentient, hovering before Kepler and Lilith.

Lilith's breath caught at the painful beauty of that solitary tone. "It's...mournful. Like a mother watching her children stray into peril." She shook her head in wonder. "Even for us - the race defiling their world."

Kepler's heart was full to bursting. With care and time, perhaps their two kindred could nurture understanding between them. But time was in short supply now.

He addressed the wavering pillars respectfully. "You know why we have come. We seek...to make amends, for our trespass." Kepler faltered, grasping for an approach that resonated with both species.

"There are those like Voronin who see only conquest and wealth. But many more wish to walk gently, learning Cronus' wisdom." He hesitated. "We lack your mastery of space and time, yet are ready to listen."

A rippling aura like aurora wove between the pillars as they conferred. Lilith held Kepler's hand tightly, knowing their lives hung on understanding the ineffable meanings in the Council's display.

The rippling ceased. In unison, the auras shifted to depict...Orion's fleet, then reformed as a single glowing pillar before disintegrating ominously.

The message was clear - turn back, or share the fate of Voronin's armada. This was the Cronians' final offer. Their rage seethed beneath like magma, halted only by delicate trust in Kepler's pleas for humanity.

Kepler blinked back helpless tears, overcome by the Council's mercy even for undeserving souls like Orion. Perhaps that capacity for grace flowed both ways between their kin.

He bowed deeply. "We accept your judgement with humble thanks. None shall threaten Cronus again while we draw breath."

A melodic tone answered, at once resonant with sorrow and hope. Then the entities began to disperse in swirling motes of light. But two broke away, approaching Kepler and Lilith directly through the lucent trees until they shone bright before the awestruck humans like captive stars.

One reached forth with a delicate aura tendril and touched Kepler's forehead gently. His mind seemed to dilate, suddenly beholding the strange intricacies of alien thought, poured forth in sympathy and expectancy.

When Kepler recovered his senses, eyes wet with revelation, the pillars were gone. Only a lingering harmonic resonance remained, ringing between Cronus and Earth.

Lilith embraced him fiercely, knowing the full meaning of the gift they had received - time, infinitely precious, to transform humankind's fate. By some miracle, they walked still as ambassadors on the galactic stage.

Chapter 19 - "Earth's Concerns"

Chapter19

0:00

/3:55

1×

Kepler sat anxiously in the communications room aboard the Cronus research outpost, rapidly calibrating the quantum transmitter to contact Earth. Beside him, Lilith put a reassuring hand on his shoulder as the display finally blinked ready.

He hesitated only a moment before opening a priority channel straight to the Global Authority's situation room. This was a call Kepler prayed he would never have to make - entreating Earth to reign in one of its most powerful corporations before an interstellar incident exploded beyond control.

The flickering holoscreen resolved into the grave faces of the Secretary-General and her advisers. Kepler saw confusion, then dawning unease in their expressions.

"Dr. Arden, explain this unscheduled call," the Secretary-General said sharply.

Kepler steadied himself. "Madam Secretary, Orion Voronin has launched a full retaliation fleet to Cronus against an alien intelligence here whose powers surpass anything we've encountered."

He quickly summarized the clash with the Cronians over Voronin's ruthless mining, the planet-wide technological breakdowns, and the begrudging restraint the entities had shown so far against mankind.

"But if Voronin attacks them directly, their retaliation could spread beyond Cronus, severing Earth's interstellar infrastructure in ways we can't imagine," Kepler implored. "You must stop him before we lose any chance of reconciliation!"

Alarm spread through the room at this revelation. The Secretary-General glanced at her advisers in dismay before turning back to Kepler.

"Doctor, if what you say is true, Voronin may have overreached dangerously. But he holds authority on Cronus." She massaged her temples wearily. "We cannot simply overrule his entire operation..."

Kepler's heart sank, but he pressed on. "Madame, I beg you. Send an emergency recall for Voronin's fleet, or we will face unthinkable consequences."

The Secretary-General held Kepler's gaze for a long moment before sighing regretfully. "I will discuss options for a diplomatic solution. But we can make no guarantees. This recklessness is of Orion's doing."

The transmission winked out, leaving Kepler and Lilith alone again under the shroud of Cronus' violet sky. He slammed a fist down in frustration.

"They don't understand the threat facing us all! What can we possibly do?"

Lilith embraced him comfortingly. "We've done what we can for now, Keph. All that remains is continuing to show the Cronians humanity's better nature until this passes."

Kepler rested his head against hers. "I fear Orion will doom us all before wisdom prevails."

Outside, the wind seemed to rise in untranslatable alien voices - screaming out against the injustice seeded on their world, yet somehow still holding back planetary vengeance as diplomacy was sought.

Lilith kissed Kepler softly. "Have faith. Whatever comes, we will face it together."

Hand in hand, they stepped outside to watch the gathering darkness and light. The future wavered between salvation and catastrophe, hinging on Voronin's pride and Earth's conscience. Their own small lives seemed to shrink until they were motes adrift between titanic forces. Yet even motes could shift balance. Until the very end, Kepler and Lilith would stand steadfast as ambassadors caught in the churning wake of civilizations colliding.

Perhaps those innocent of Earth's sins could walk blameless before the Cronian Council. Kepler clung to that fragile hope, though it seemed ever more beleaguered as warships streamed skyward like thunderheads of doom.

Chapter 20 - "Apocalyptic Visions"

Chapter20

0:00

/4:20

1×

Darkness engulfed Kepler as he drifted into an uneasy sleep, the threat of catastrophe looming over both Earth and Cronus keeping rest at bay. He had done all he could to avert disaster, yet peace still seemed beyond their grasp.

As slumber finally embraced him, Kepler found himself adrift in an utterly lightless void, strangely similar to the sensory deprivation of quantum travel. But there was no ship to shelter his body here, just his conscious mind floating disembodied through the endless night between galaxies.

Gradually, pockets of nebulaic gas drifted into view, tinged with the crimson of Cronus' giant sun. Kepler tried to regain his bearings, but the space around him seemed distorted, obeying an alien geometry.

Without warning, the gas clouds coalesced into violent stellar nurseries, birthing new suns in explosive fury. Their flaring brilliance seared Kepler's senses. As the newborn stars shone brighter, space itself seemed to ripple and dilate around them - and through the shimmering celestial mirages, Kepler glimpsed titanic shadows moving...

Heart thrashing, he desperately sought the familiar comfort of stars and constellations. But there was only the angry glare of the nebula and those colossal shapes stirring languidly in the distance.

As Kepler stared, frozen by the awful sight, a crooning resonance echoed across the lightyears. The Cronians! Their mental presence enfolded Kepler gently, emanating profound melancholy. Weeping for worlds beyond knowing, now facing ruination...

The visions intensified, stars and worlds blurring into streaks of light as Kepler hurtled across the Milky Way faster than signals could carry. He beheld vistas of civilizations both familiar and unimaginably alien, linked by glittering networks of quantum trade routes. Yet all shared the same primal seed - life, taking root and flowering in manifold ways Kepler could barely comprehend.

Now darkness loomed over those far-flung yet kindred worlds. A shadow falling not just on isolated Cronus but over all that had kindled from the ancient triumph of order over chaos. Everything Kepler ever dared dream across his decades of wandering the cosmos - all that promise, delicate as snowflakes, teetered on the crumbling edge of extinction's merciless gravity well...

Kepler cried out as the visions pierced his mind sharper than any physical agony. He was a mote watching creation unravel around him, all that joy and ingenuity meaningless against forces that operated on inhuman timescales.

Just as oblivion rushed to embrace Kepler completely, the Cronians surrounded his mind gently, filtering the psychic torrent until he could bear its full meaning. The shadow was not yet absolute - there was still hope, while even one innocent soul stood to shield the light.

The visions dissipated reluctantly, leaving Kepler alone in the abyss once more. But the heavy sickness of dread had lifted from his spirit. He sensed the dreamscapes were a gift, chilling yet necessary, to steel his resolve for the coming tribulation. A reminder that life transcended any single epoch, however dark.

As waking's warm light enveloped him, Kepler knew what must be done. Every civilization was the custodian of those to follow, bound in cosmic kinship across time. Though Orion rejected his responsibilities, Kepler would remain true to that eternal charge.

He rose renewed in purpose, rejoining Lilith to await Voronin's fateful arrival. Though a long night loomed, hope yet remained while the Cronians showed faith in humanity's redemption. Kepler focused on that lone bright horizon beyond despair. If these were to be their final days, he and Lilith would face the end standing sentinel together, two small souls defying the dark between galaxies.

Humanity would confront its reflection soon in Cronus' crystalline gaze - would they see gods or monsters looking back? At this crossroads between ruin and redemption, all Kepler and Lilith could do was keep holding up the mirror, unflinching against the approaching storm.

Chapter 21 - "Alien Partnership"

Chapter21

0:00

/3:44

1×

The violet mists of Cronus swirled around Kepler as he sat deep in meditation, attempting to reconnect with the alien presences inhabiting this strange world. Since the visions, his mind felt altered, as if permanently receptive now to the Cronians and their unknowable ways. He hoped desperately they might sense his desire for reconciliation.

Time was evaporating faster than worlds could form; Orion would be here in mere hours. Yet the Cronians had shown astonishing restraint against mankind so far. Were they still open to nonviolent solutions, if Kepler could kindle trust and understanding between their disparate kin? He focused inward, transmitting thoughts of cooperation and goodwill across the unseen gulfs separating their species.

At first, only silence answered. Then, slowly, a resonant tone began to rise among the towering crystal trees surrounding the grove where Kepler kept his patient vigil. The harmonic vibrations were at once utterly alien yet strangely familiar, evoking complex emotions not translatable into any earthly vocabulary. Kepler sensed curiosity, surprise, cautious optimism and something deeper—a fragility not found on physical planes of existence.

As the resonances ebbed and flowed, colors and fractal patterns swirled into being before Kepler's astonished eyes. The Cronians were engaging, sharing concepts and history at a rate human minds could scarcely grasp. Like psychic gulfs opening around nebulous islands of meaning, the discourse defied verbal description.

Kepler surrendered himself fully to the reality-warping exchange, set adrift on a sea of alien thought. Flickers of memory and emotion not his own washed over him as the entities explored his subjective realm with delicate precision. Through what Kepler could only perceive as eons of debate and revelation between their species, he struggled to impress his memories of humanity's virtues—sacrifice, love, yearning for connection.

When the communion finally withdrew like a falling tide, Kepler collapsed in utter exhaustion. There were no words for what had passed between them, but he sensed a fragile bridge had been forged. Perhaps now the Cronians grasped that mankind was more than just Voronin's cruelty and greed. That there were yet innocents striving to walk in balance across the stars.

As Kepler lay bathed in prismatic starlight, a single harmonic tone resonated through his mind, at once resonant with melancholy yet pulsing with newfound hope. The meaning was clear—we acknowledge your integrity. Further violence will be withheld...for now. A future alliance is conceivable.

Joy suffused Kepler's weary spirit. Trust remained precarious, the threat still loomed ahead. But in their selflessness, the Cronians had found a seed of possibility in human nature that Kepler vowed to nurture. They would face Voronin's incoming armada together, kindred souls holding back the tide of destruction.

There were no miracles or absolutes, only tesseract paths still unfolding. But Kepler had bought them time and chance. Perhaps that would be enough to steer both races away from the brink. Eyes fixed on the shimmering alien sky, Kepler allowed himself an exhausted smile. Sleep could wait—this newfound kinship was too fragile and strange for rest. Together they kept their patient watch, two souls standing against the dark.

Chapter 22 - "Earth vs. Orion"

Chapter22

0:00

/5:39

1×

A tense silence filled the Global Authority's situation room as Secretary-General Veda Patel ended the emergency session regarding Orion Voronin's Cronus fleet. Despite Dr. Arden's dire warnings, they had failed to reach a consensus on stopping Voronin's imminent retaliation against the entities on Cronus.

Powerful corporate interests tied to Voronin's Starfire Corporation had vehemently opposed any direct action to hold Orion accountable, protecting their own profits and leverage. Arguments went back and forth for hours, growing more heated as the fleet neared its estimated arrival at Cronus.

In the end, Secretary-General Patel lacked the political capital to unilaterally decree Voronin stand down. His family's dynasty and corporate empire wielded nearly sovereign powers across many colonies and outposts. Attempting to restrain him could fracture Earth's already precarious interstellar unity.

As Kepler’s transmission winked out, Patel sighed heavily and dismissed the exhausted cabinet members. She understood Kepler's urgency all too well - the lives of billions likely hung on what the Cronus fleet encountered. Perhaps even Earth's survival if the entities there retaliated in full.

But was that worth rupturing humanity's fragile post-diaspora politics? Patel's pragmatism warred with her conscience as a leader. Sometimes holding civilization together meant hard compromises.

Alone now, she stared out the panoramic window overlooking gleaming Geneva, ancestral heart of Earth's global governance. She swiped open a secure channel to speak with the one person who might prevail where she couldn’t – Orion’s father.

Dimitri Voronin's weathered but keen face flickered into view. Despite his age, the Voronin patriarch was still fiercely lucid.

"Mr. Voronin, thank you for taking my call. We find ourselves in an urgent situation regarding your son.”

Dimitri raised a hand. "No need to elaborate, Madame Secretary. I have already heard Orion's foolhardy plans." His expression was grave. "We Voronins often rush ahead in pursuit of greatness, blind to consequences. But I know my son has a conscience under his ambition. I will personally speak with him before it is too late."

Patel nodded in relief. "You may be the only one he still heeds. I pray you can make him understand the true stakes."

Dimitri gave a sad smile. "The dreams of my generation die hard, Madame Secretary. The cosmic destiny we promised our children now burdens them greatly." His eyes were distant, seeing past regrets. "But Orion's fate is not yet sealed. There may be a path to turn him from our tainted legacy."

The transmission ended, leaving Patel pensive. Perhaps there was hope of averting disaster, if Kepler's warnings could rekindle Orion's humanity. She sent up a silent prayer - for Kepler and Lilith's safety, for the beings facing Voronin's wrath, and for the soul of one ambitious man whose choices bore immense consequence.

Aboard the Cronus flagship Colossus, Orion stood alone on the command deck, steeling himself for the battle ahead. He had sacrificed too much, come too far along the lonely path to glory to turn back now. His was a sacred duty - to spread humanity's light across the uncaring cosmos.

So why then this hollow sickness in his chest, this doubt creeping through his thoughts? Orion pushed it fiercely down. He would do what must be done for Earth's prosperity, as all Voronins had. But his hands trembled of their own accord as the fleet streaked towards its fateful rendezvous with Cronus. There, shattered trust would unleash terrible forces, perhaps even world-ending. Yet still Orion continued on the ordained course, unable to conceive another way.

When the priority alert chimed unexpectedly through his neural implant, Orion felt cold certainty - this was no ordinary communique. He opened the channel with a deep breath. An aged but imposing figure materialized before him in crackling blue light - Dimitri Voronin, patriarch of their dynasty. Orion's father had not spoken to him in years, since the rift over Orion taking control. He tensed, bracing himself.

"Orion, my son. We must speak." Dimitri's tone was weary, but intense in its sincerity. "Turn your fleet from this reckless path while time remains. For the sake of all Earth, and your own soul."

Orion stood paralyzed, childhood insecurities welling up to war with his resolute ambition. In the shivering balance between futures, everything hung on the choice of one conflicted man - oblivion or deliverance for two civilizations.

Humanity's fate remained bound to Cronus' crystalline shores. What needed to break now was not alien worlds, but Voronin's iron chains to the past.

Humanity's fate remained bound to Cronus' crystalline shores. What needed to break now was not alien worlds, but Voronin's iron chains to the past. In his ambition, Orion struggled to see beyond the confines of his forebears' legacy. But confronted now with the gentle wisdom of the father he had defied, Orion felt the first cracks forming in his fiercely held convictions.

There was still a chance to avoid calamity, but only if Orion relinquished old grudges and fears. Standing at destiny's fulcrum, he faced the hardest question - were his dreams worth the price of Earth's soul? In the void between stars, the battle for one man's conscience raged on.

Chapter 23 - "Quantum Warfare"

Chapter23

0:00

/6:16

1×

As Orion's fleet emerged from quantum jumpspace, Cronus loomed before them - a magnificent sapphire orb veiled in violet mist. For a heartbeat, Orion faltered, doubt and longing welling up in his chest. Was the dream of his forebears worth this coming bloodshed?

But he forced down the weakness, hardening his heart. Too much sacrifice had led them to this precipice. He would see their cosmic destiny fulfilled, no matter the cost.

"Battle stations!" Orion commanded. Across the flagship Colossus, klaxons blared as the armada readied itself. Cronus seemed to shudder in response, its placid clouds gathering into broiling thunderheads.

Kepler watched the standoff from the surface, clinging to Lilith in despair. "He's really going to attack them..."

Around them, Cronus' strange harmonics rose in a dissonant chorus - whether lament or battle cry, none could tell. The Council convened as shimmering pillars within the crystal grove, their shapes flickering rapidly in agitation. Kepler prayed they would show restraint, but feared Orion was about to tear the fragile trust between them apart forever.

With blaze of fusion drives, the first waves of fighters streaked towards Cronus' atmosphere. But as they neared the planet's lambent clouds, reality itself seemed to warp around them. Quantum static engulfed the ships, overriding systems and dragging them down like sinking stones. Through sheer kinetic force, a few impacted the surface, gouging out flaming craters.

Orion gripped his command throne in disbelief. "Report! What happened?"

"Some kind of...spacetime manipulation," his sensor officer stammered. "Our ships lost all propulsion and integrity fields."

Rage kindled in Orion's eyes. "Deploy the Cronium-shielded reserves! We'll see how these apparitions stand against our might!"

From specially retrofitted hangars, the next assault wave dove towards Cronus behind layers of exotic metallic shielding. They punched through the atmosphere and began strafing dazzling beams across the planet's surface, setting alien forests ablaze.

The Council's pillars flashed urgent red, resonating with the anguish of the dying biosphere. Then suddenly they vanished, dispersing into a billion motes of thought that permeated the quantum planet-mind.

Up in orbit, navigational systems started glitching inexplicably, pilots seized by sensations of unfathomable entities brushing against their consciousness. Like the peripheral vision of nameless phantoms, the Cronian intelligences infiltrated Orion's ships and crews.

Chaos reigned across the fleet as reality itself turned traitor. Cronium shields flickered uselessly as ships lost integrity, keeling out of formation in eerie silence before being consumed by the roiling clouds.

Orion could only watch helplessly as ghostly phenomena ran rampant through his mighty armada. But retreat was unthinkable. He strengthened firewalls around his consciousness, rallying the remnants of the fleet.

"Full bombardment, give them hell!" he thundered. From the Colossus' spinally mounted siege cannon, a colossal beam of energy lanced down, boiling away kilometers of atmosphere and landscape. Orion gritted his teeth against the photon forces, consumed now only with obliterating these alien monstrosities, whatever the cost.

On the scoured surface, Kepler and Lilith took shelter against the orbital onslaught, their minds reeling withCronus' psychic death cries. They clung together, innocent hearts breaking at the loss engulfing both species.

"Kepler..." Lilith spoke urgently, eyes blazing with revelation. "The resonance...feel it?"

Beneath the bombardment, Cronus was trembling not just in pain, but from vast forces building in response. The Council had awakened the planet's deepest quantum foundations. In self-preservation, a fierce intellect a billion years older than mankind was stirring.

And as Kepler attuned his psyche to Cronus' anguish, he glimpsed the planet-mind's pure, crystalline thought unspooling across spacetime itself - evaluating Orion's flock of metal insects...and preparing counter-judgement.

"No! They mustn't!" Kepler cried out. "Spare them, I beg you!" But Cronus was already passing the verdict, the collective psyche of eons marshaling its strength against the invading flea.

In the skies overhead, the Colossus shuddered as arcing discharges engulfed its hull. Alarms blared as maglocks disengaged and bulkheads breached. The dreadnought was coming apart at the seams.

Orion stumbled on the command deck as reality dissolved around him. But through the chaos, he recognized the sequence flashing on his console, awaiting final judgement. The activation code for the Colossus' neutrino lance - a weapon to crack open worlds.

Cronus filled the viewport now, defenseless below. Orion's finger trembled above the console. After everything, one word from him would blast this alien world into cosmic history. His birthright within reach, if he dared claim it.

Yet as Orion wavered at destiny's precipice, Kepler's appeals echoed through his mind. Was transcendence worth becoming the butcher of thinking worlds? In this final crucible, the man must overthrow the conqueror.

With a scream of anguish, Orion slammed his fist onto the override, aborting the colossal charge building within the flagship. The deck bucked as the Colossus spiraled brokenly into the atmosphere, shedding its weapons in fiery contrails.

Aboard the plummeting hull, Orion drifted in zero-g, sobbing silently amidst the screams of torn metal. Everything he had fought for, gone. But in its ruin, perhaps something nobler could be salvaged from his family's legacy.

With a seismic groan, the ravaged Colossus came to rest in a valley of violet Cronian flora.

Chapter 24 - "Reconciliation"

Chapter24

0:00

/3:32

1×

In the wake of Orion's doomed armada, a tense calm had settled over Cronus. The planet's surface remained unsettled and volatile, reacting to the disturbance across its quantum depths. Pockets of unnatural decay marred the alien wilderness Kepler had come to love, like infected wounds slow to heal.

The research outpost was in lockdown, communications crippled and supply ships unable to approach past the blockades imposed by the Cronian Council. Inside its sterile walls, a dispirited gloom had taken root as the stranded team waited helplessly for salvation or judgment.

Kepler spent his days in the crystalline grove, keeping a patient vigil and trying to re-establish contact with the entities who walked this strange land. Lilith stayed by his side, a stalwart companion on this lonesome vigil. The Cronians remained elusive, the harmonics that once resonated so potently now dimmed to a whisper teasing the edge of perception.

"Do you think they mourn?" Lilith asked softly one twilight, imagining the great pillars of light bowed under the weight of remorse.

"In some fashion we cannot conceive," Kepler replied. "But I believe there is hope of reconciliation between us still." He rested his head wearily on her shoulder. "We only need to rebuild the trust that Orion shattered."

As Cronus' violet moons rose huge and luminous above the horizon, Kepler felt a strange resonance begin to build within his mind. The Council was stirring, reaching out across the gulf of otherness. He took Lilith's hand tightly as a familiar presence enfolded his thoughts. Strange concepts blossomed like fractal flowers, painting impressions of sadness, hope, expectation. Of waiting for their small ambassador to attend the conclave once more.

Kepler's heart leapt even as he trembled. The Cronians had spared earthly lives where they could, showing mercy even amid vengeance. But to stand again in their phantom court required courage beyond reason.

Lilith sensed his hesitation. "Go to them, Keph. Their trust in you is all that remains between our races now. I will be right here, always."

Steeling himself, Kepler surrendered to the psychic pull. His perceptions blurred, coalescing again within a landscape at once crystal and ether. The Council ringed this manifold space, radiating expectancy. With infinite care they reached out, joining Kepler and through him humanity's essence into their congress. What passed between their minds in the hours beyond flowed space and words.

When Kepler finally awoke in the muted dawn, he wept with profound catharsis. Across the impossible gulf, understanding had taken fragile root. The Cronians would continue their guardianship of this wounded biome, but allow humanity's return once they had walked the inward paths of remorse and wisdom. There was bitterness and hardship yet to come. But past the coming darkness, a possible future now beckoned where two kindred intelligences might heal and share each other's light.

Lilith smiled through her own tears, knowing what had passed in that unearthly conclave. No longer lost, they walked again on hallowed ground, missionaries of life itself. In trust there was hope. And in hope, the faith that life yet defeated death under whatever garb it donned across the eternal cosmos.

Chapter 25 - "New Dawn on Cronus"

Chapter25

0:00

/4:17

1×

A soft azure dawn crested over Cronus' violet forests, casting everything in luminous hues. Kepler paused from tending his alien garden to admire the daily miracle of light reborn. Even after twenty years here, the beauty of this adopted world still stirred his weary spirit.

"Father! Come quickly, the phasedancers are out!" Kepler turned with a smile at his daughter's voice. Nineteen-year-old Malika beckoned excitedly from the porch of their sprawling alien home, woven from living Cronian trees. Her eyes, one blue and one hazel, shone with youthful curiosity, reminding Kepler of himself long ago.

He joined her there, gazing out at the clearing beyond their dwelling. Sinuous forms spun hypnotically above the mosses - the phasedancers, a quasi-corporeal entity native to Cronus. As the first dawn rays struck them, their bodies fractured prismatically into quantum ripples, casting dazzling rainbows across the morning mist.

Malika watched, enraptured. Though born human, she was as much a child of Cronus as Earth. Kepler and Lilith had raised her in close harmony with this strange world and its mysteries. She represented a new generation who might someday bridge the lingering divide between their species.

The phasedancers dissolved reluctantly back into motes as the sun rose higher. Lilith emerged just then from the house, pulling on her boots. "Well, this Council meeting won't attend itself," she said wryly. Though there was still much hurt to heal, the Cronians had kept their promise, allowing humanity's gradual return. Her engineering expertise remained invaluable to maintaining the delicate cooperation.

Malika turned to them eagerly. "Mother, Father - may I attend too? I long to see the Council again." Kepler considered a moment, then nodded. "Just this once, starlight. You're old enough now to join the discourse, if you observe proper courtesy."

The crystalline grove shimmered just beyond the ridge from their home. Kepler paused at its edge, allowing the familiar harmonic tones to wash over him. Though the Council had forgiven, they did not forget. Respect must be shown.

Taking Malika's hand, he led her into the radiant copse. The Cronians manifested slowly as pillars of light, colors rippling in delicate patterns along their formless surfaces. Malika gasped softly at their austere beauty. Through years of tutelage, Kepler knew the nuances of posture and frequency that conveyed friendship, trust, humility to these beings. He shared what cues he could with Malika through subtle gestures.

As the discourse began, resonances ebbing and flowing, Kepler sensed the entities' curiosity about this newcomer who represented both species. With infinite care, they opened pathways for Malika to share her memories and dreams, coaxing forth her shy but earnest hopes for unity. Moved by the insight shared, the Council resonated approval, wreathing Malika in light. They too heard the future's call - of understanding nurtured through the open spirit of innocents.

Afterwards, as they turned homeward, Malika chattered with joy at being welcomed into the alien conclave. But Kepler remained pensive. The Council's gift lay heavy in his pocket, a delicate crystal lattice pulsating with hypnotic inner fires. A seed of wisdom, offered to humankind should they choose to grow it. But much bitterness lingered still. Would it be planting in fertile soil or stony ground? The choice was not Kepler's alone to make.

Back home, Lilith greeted them with news - a priority transmission from old comrades back on Earth. The first thread tied anew. Kepler's hand closed over the warm crystal, praying its alien light might guide lost kin. There was hope still, fragile as any new dawn. If darkness came, they would face it together, planting seeds for mornings yet to come. For now, it was enough.

Chapter 26 - "Kepler's Reflections"

Chapter26

0:00

/5:01

1×

Kepler sat alone on a rocky outcropping overlooking the Cronian sea, the twin moons casting rippling emerald light across the gently undulating alien waters. He inhaled the crisp salty air, feeling the breeze caress his weathered features. At 70 years old now, Kepler could sense the long decades behind him, written in furrows across his brow and grey streaking his once dark hair.

How quickly the time had flowed, like cosmic streams, thought Kepler wryly. As a young man he had set forth to distant stars with fiery curiosity, lusting after new worlds and undiscovered knowledge. Never realizing that what he sought was not to be grasped, but gradually unveiled through patience and humility.

Now at the twilight of his years, Kepler found himself returning often to Cronus and its violet shores. The planet that had almost shattered mankind's interstellar reach, yet became the instrumental anvil upon which human civilization hammered out a new maturity.

Things were far from perfect, reflected Kepler soberly. Humanity's rapacious appetites and tribal divisions still threatened the unity so essential to survival. But Orion's humbling had ignited at least a thirst for temperance and conscience, guiding explorers to walk gently amid the splendor.

The Cronians too had evolved since that first fraught contact. Their mechanistic quantum manipulations were yielding to more harmonious resonance with the galactic forces they channeled. Kepler had taught them much of non-linear consciousness, and they in turn imparted wisps of eternity's perspectives to him.

Together both races now nurtured the cosmic code at the heart of sentience. And in the young like Malika was kindled a new spirit of exploration - one traveling inward as much as outward. Not to grasp or conquer as the simple minds of Kepler's era had sought, but to share, support and celebrate diversity.

It gladdened Kepler profoundly to see Malika taken under the Council's wispy tendrils, treated as one of their cosmic kindred. Perhaps her generation would at last heal the rifts between worlds that had nearly swallowed his whole in the dark.

The icy void always beckoned with indifference, yet now Kepler sensed fragile bonds woven across the abyss - gossamer threads of courage, curiosity and empathy. While those precious ligaments endured, civilization yet carried worth and moment.

Kepler ran his fingers across the smooth Cronian crystal he now wore around his neck, resonating gently with the planet's quantum heartbeat. He had lived through wondrous and terrible ages alike. Seen brilliance laid low by arrogance, only to rise again brighter and humbler. Watched societies flourish and lapse on a thousand worlds. Measured his small life against the uncaring sweep of galaxies.

Through it all, one truth alone endured - that they all danced together, however disparate their forms and natures. Life's shared notes reverberated through the endless night, binding in strange harmonies. To foster that music was Kepler's legacy, and he rested content now in its fragile echo.

The years ahead remained to be penned. But Kepler carried little worry for what fates they held. A patient wisdom flowed in him, rippling out endlessly through the lives he had touched even fleetingly. The tale passed beyond any single teller.

With a quiet smile, Kepler rose and turned towards the beckoning lights of the settlement where Lilith awaited. He paused for one last lingering look across the lucent alien sea, holding close the wisdom gleaned. Then he set forth again in contemplation of the journey ahead. The path stretched on, winding and continuous as time's own flow.

Humanity's course was unclear, but Kepler had woven his strand in destiny's tapestry. Wherever the coming generations roamed under distant suns, something of him would echo in their communion with the infinite. For beyond cold eternity burned fellowship's warm light, kindled in pondering the heavens together. All who dared lift their eyes to the uncharted abyss were kindred in that cosmic flame.

In its glow, Kepler had glimpsed the braiding of life itself - resilient and unexpected, but ultimately synchronized by code deeper than form. Again and again, its flowering persisted through all travails. And in that endurance Kepler saw the universe unveiled not through dominance, but participation in its immortal spirit.

So long as that bright spark yet lingered against the darkness, there was cause for hope. With a lightened heart Kepler went to join the dance, a small but rhythmic step across the eons. The ancient pulses guided him home...

Six months ago, I created a collection of imaginary creatures with the help of GPT-3 and Dall-e 2. With the rapid advancements in generative AI, it's now time to dive back into the world of these imaginary creatures with a fresh perspective, using Midjourney 5.

This time, I wanted to approach the project as if I were on an expedition to an unexplored planet, armed with just my AI camera and an insatiable curiosity. And so, I began my journey into the unknown, eager to discover the fascinating creatures that lay ahead.

I hope you enjoy discovering these creatures as much as I did.

I found myself pondering the astonishing capabilities of the latest iteration of OpenAI's language model, GPT-4. As someone who has been following the development of these models with avid curiosity, I couldn't help but wonder: How can we really test if this model has some kind of general intelligence?

After a few attempts to come up with interesting questions that I could answer but GPT-4 could not I decided on a new direction: old school text adventure games!

I decided to put it to the test in an interactive text adventure game called "The Dreamhold". The thought of a machine not only playing the game but also reasoning about it seemed almost fantastical. Nevertheless, I decided to dive headfirst into this experiment.

My goal was to see whether the model could not only play "The Dreamhold" but also provide a coherent explanation of its thought process and decision-making.

As the game progressed, I found myself continuously amazed by the model's ability to navigate the intricate labyrinth of "The Dreamhold", uncovering hidden paths and deciphering enigmatic clues along the way. With the excitement of an adventurer exploring a mysterious realm, I eagerly observed as GPT-4 ventured through the game, growing increasingly astounded by its capacity for reasoning and problem-solving.

Finally, as the experiment drew to a close, GPT-4 presented a comprehensive overview of the rooms it had explored, as well as the uncharted paths and unsolved mysteries that still awaited further inquiry. It is crazy to think how fast this technology developed! GPT-4 had not only played "The Dreamhold" but had also demonstrated a remarkable ability to reason about the game and its various elements. As we continue to push the boundaries of AI and explore its seemingly limitless potential, I can't help but wonder how this will change the world in the coming years.

Here is the complete transcript of GPT-4 playing "The Dreamhold" (up to the 8000 token limit).

• Text in green is "user input". These are my questions and statements, to test if the model has an understanding of what it is doing.
• Text in gray is the "game output", this is what the game outputs, directly copy pasted to GPT-4.
• Text in yellow is generated by GPT-4. These commands are given directly to the game.
• Text in purple are remarks, that are not used as input to the model.

Part 2: The journey continues past 8k tokens

Because it was getting close to the 8k token limit, I asked GPT-4 to summarize the progress so far. It gave the following answers, and even created a script to output the rooms explored so far as a DrawIO diagram in XML.

Then I started a new conversation and asked it to continue the game, given this summary.

Part 3: Planning for the future

We were getting close to 8k tokens again, so I asked GPT-4 to summarize its progress in the game again.

Then using this information, I asked it to make a plan for advancing in the game and retrieving the third point.

Welcome to the Museum of Imaginary Art.

A playground for AI creativity, this museum is home to a diverse array of artworks created entirely by computers and artificially intelligent systems. Here you’ll find works that push the boundaries of what is possible with computer-generated art.

I am the curator of this museum. The stories for the artists and descriptions of the expositions are all generated by GPT-3 (text-davinci-003). The artworks are generated by Midjourney based on these descriptions. My role is to imagine how these artists would think, and select the works I think they could have created.

On the walls of the museum, you will find a variety of artworks. From swirling abstracts to intricate landscapes, each piece is unique and offers a new perspective on the world.

I hope that you enjoy your visit to the Museum, and that you leave with a greater appreciation for the creativity that AI can bring to the world.

Artists on display:

• Sue Field - The power of imagination (expressionist painting)
• Sarah Johnson - The beauty of nature (abstract landscape painting)
• Nelson Rougeau - The Beauty of Uncertainty (digital art)
• Adora H - In between (photography)
• Mur Adzio - Meta Illusions (abstract / surrealism painting)

Sue Field - The power of imagination

Sue Field specializes in abstract expressionist paintings. Her artworks are composed of impacting colour fields, dynamic brushstrokes and a unique, energetic energy that captures the essence of the modern world. Her art transports the viewers to a realm of abstraction and emotion while maintaining an aura of confident self-expression.

Sue Field grew up in a small town in the countryside, surrounded by the beauty of nature. From a young age, she was drawn to the imaginative potential of art, and spent hours in her studio experimenting with different mediums and styles. After studying art in college, she moved to the city to pursue her dream of becoming a professional artist. She quickly gained recognition for her bold and expressive abstract expressionist paintings, which captured the energy and emotion of the modern world.

The exhibition titled “The Power of Imagination” is centered around the idea of creating our own realities through the imaginative power of our minds. Sue Field’s artworks capture the pure potential of the human imagination, expressing the potential that lies within each of us. This exhibition is an ode to the creative power within us all, and an inspiration to viewers to explore the possibilities that their imagination can offer.

Sarah Johnson - The beauty of nature

Sarah Johnson is a talented and passionate artist who creates abstract artworks by taking inspiration from nature. Using a combination of oil paints, charcoal, and mixed media, her artworks bring to life a stunning landscape full of detail and texture.

Sarah Johnson grew up in a family of artists, and spent her childhood exploring the natural beauty of her home in the mountains. She developed a deep appreciation for the diversity and complexity of the natural world, and began to study botany and ecology in order to better understand the ways in which living things interact with their environments. As an artist, she combines her love of nature with her passion for creativity, using a range of mediums to create abstract landscapes that capture the beauty and wonder of the natural world.

The theme of the exhibition is “The Beauty of Nature”. Sarah hopes to capture and convey the beauty of the natural world, from majestic mountains to wild seas, and everything in between. Through her works, Sarah strives to help viewers appreciate and connect with nature.

Nelson Rougeau - The Beauty of Uncertainty

Introducing an exciting, new and unknown digital artist called Nelson Rougeau, who has developed a novel digital art style. His artworks are composed of complex, multi-layered visuals that express his own personal vision of the world. His works are at once abstract and figurative, and often evoke a dream-like atmosphere of surrealism and fantasy.

Nelson Rougeau was always drawn to the world of art and technology, and spent his teenage years tinkering with computers and experimenting with digital art. After studying computer science and art in college, he began to develop his own unique style of digital art that explored the beauty of uncertainty and the ways in which technology can enhance and transform our perceptions of the world.

The theme of the art exhibition is “The Beauty of Uncertainty”. The artist is exploring the idea of how beauty can be found in moments of uncertainty and instability. He uses his artworks to create a visual representation of the beauty and chaos of life, and to express the idea that uncertainty can be embraced, accepted and celebrated.

Adora H - In Between

Introducing an exciting, new and unknown fine art photographer called Adora H. Her work has a strong, minimalist approach and is characterized by her use of light and shadows, which she captures in a contemporary, modern visual style. Her work focuses on simple elements such as light, reflections, and shapes to create powerful and evocative photographs.

Adora H was born and raised in a small town in the countryside, where she spent her days exploring the fields and forests with her camera. From a young age, she was drawn to the beauty of the natural world, and developed a keen eye for capturing the details and nuances of the landscapes and wildlife that surrounded her. After studying photography in college, she moved to the city to pursue her passion for photography, and quickly gained recognition for her evocative and thought-provoking images that explored the theme of "in between" - the spaces between objects, people, and ideas.

This exhibition is named "In Between" and reflects Adora's exploration of the liminal spaces between the real and the imaginary, between light and shadow, between the seen and unseen. Through her art, Adora captures moments of beauty and serenity amidst the chaos and uncertainty of life. She invites viewers to explore the hidden depths and nuances of the world around them.

Mur Adzio - Meta Illusions

We are pleased to present the first-ever art exhibition by the unknown artist, Mur Adzio. An abstract painter inspired by the surrealist movement of the early 20th century. They explore the wonderfully strange images of an illogical universe, delving into optical illusions that challenge our ideas of reality. Their artwork combines vivid and vibrant colors to create unique scenes that inhabit a land beyond logic.

Mur Adzio grew up in a bustling metropolis, surrounded by the hustle and bustle of city life. From a young age, he was fascinated by the complexities of the world around him, and spent his days exploring the streets and alleys of the city, observing the ways in which people interacted with their surroundings. As an artist, he began to explore the concept of "meta illusions" - the illusions that exist within other illusions - using abstract and surrealist techniques to create art that challenges viewers to question their perceptions of reality. His art has been exhibited in galleries around the world, and he is known for his unique and thought-provoking creations.

The exhibition is called 'Meta Illusions', and explores the concept of how reality can be warped by optical illusions. Through this exploration, Mur Adzio encourages us to think and reflect on our own perceptions of what is real and what is illusion.

The transformer is a neural network architecture that was proposed in the paper "Attention is All You Need" by Ashish Vaswani, et al. in 2017. It is a powerful architecture that lead to many state-of-the-art results in the last few years. It can be used to generate text (GPT-3), create beautiful images from text (Imagen, Dall-e 2, Stable diffusion), compose music (music transformer), speech-to-text (Whisper), understanding protein structure (AlphaFold), teaching cars to drive themselves (Tesla FSD) or even learn to do many different tasks (Gato). There are probably many amazing results I forgot to name, and many more to come.

Large language models (LLMs) are going to have a large impact in the future. We are probably going to interact with these models a lot in the future to brainstorm, help us with creativity, make sense of large amounts of information and maybe even understand ourselves better. GPT-3 is already helping me with coding and writing (also for this blogpost), and these models are going to get better really fast. That is why I want to understand how they work.

In this blogpost I will show you how to build a text generation model from scratch using the transformer architecture. I will show the coding process, and will try to make each step as simple as possible. The aim of this post is not to build the best text generation model, but to try to make each step of building one as clear as possible.

Overview

Text generation models like GPT-3 are built using the transformer architecture. This architecture was first proposed in the 2017 paper "Attention is All You Need".

Unlike typical sequence-to-sequence models used for translation tasks, which have both an encoder and decoder, text generation models only require a decoder. This is because the input and output sequences are essentially the same - the model generates text token-by-token based on previous context.

The decoder-only architecture looks like this:

So for text generation, we only need to focus on implementing the decoder portion. The key components we'll cover are:

• Tokenization - breaking text into tokens
• Input embeddings - turning tokens into vectors
• Positional encodings - retaining sequence order information
• Masking - preventing model from peeking ahead or looking at padding tokens
• Multi-headed self-attention - relating different input tokens
• Decoder stack - refining understanding over multiple layers
• Language model head - predicting token probabilities

By incrementally implementing each of these pieces, we'll end up with a full text generation model. I'll explain both the concepts and show concrete code for each part.

The goal is to provide an intuitive understanding of how transformers work under the hood. Let's get started with tokenization!

Tokenizing the text

Natural language processing models like transformers operate on numerical data. However, our raw data is often in the form of human-readable text. To bridge this gap, we employ a process called tokenization.

Tokenization involves breaking down the text into smaller chunks known as tokens. The granularity of these tokens can range widely: a token might be as small as a single character, as large as a word, or anywhere in between. The choice of token size depends on the specific task and method being used.

For instance, GPT-3 uses a form of tokenization called Byte Pair Encoding (BPE), which tokenizes text into subwords. These subwords are sequences of characters that frequently appear together. This approach is a middle ground between character-level and word-level tokenization, balancing efficiency and versatility.

The Role of a Vocabulary

Tokenization involves converting each token in the text into a unique integer identifier, or index, using a dictionary called a vocabulary. The vocabulary consists of a list of all unique tokens that the model should recognize.

Consider the sentence "I am a cat." If our vocabulary is ["<PAD>", "I", "am", "a", "cat", "dog", "mouse"], each word in the sentence maps to an index in the vocabulary. "I" corresponds to 1, "am" to 2, "a" to 3, and "cat" to 4. So, our sentence becomes [1, 2, 3, 4] in numerical form.

Dealing with Variable-Length Inputs: Padding

Transformers typically expect inputs of a fixed size. However, real-world text data comes in sequences of variable length. To accommodate this, we use a technique called padding.

Padding involves supplementing shorter sequences with extra tokens (typically designated by the index 0) to match the length of the longest sequence in the batch. For instance, if we want all inputs to be six tokens long, our example sentence "I am a cat." becomes [0, 0, 1, 2, 3, 4].

The size of a vocabulary depends on the complexity of the language and the granularity of the tokenization. We are going to use a simple tokenizer which recognizes just 39 tokens.

On the other hand, GPT-3's tokenizer uses a vocabulary of 50,257 tokens. This large vocabulary includes a broad range of English words, common word parts, and even whole phrases, enabling GPT-3 to understand and generate highly diverse and nuanced text.

Simple tokenizer example

We create a very simple tokenizer that can encode the characters a-z, the numbers 0-9 a period and a whitespace character. The padding token is mapped to 0.

Our dictionary is very simple, and contains only 39 tokens.

class Tokenizer:

    def __init__(self):
        self.dictionary = {}
        self.reverse_dictionary = {}

        # Add the padding token
        self.__add_to_dict('<pad>')

        # Add characters and numbers to the dictionary
        for i in range(10):
            self.__add_to_dict(str(i))
        for i in range(26):
            self.__add_to_dict(chr(ord('a') + i))

        # Add space and punctuation to the dictionary
        self.__add_to_dict('.')
        self.__add_to_dict(' ')

    def __add_to_dict(self, character):
        if character not in self.dictionary:
            self.dictionary[character] = len(self.dictionary)
            self.reverse_dictionary[self.dictionary[character]] = character

    def tokenize(self, text):
        return [self.dictionary[c] for c in text]

    def character_to_token(self, character):
        return self.dictionary[character]

    def token_to_character(self, token):
        return self.reverse_dictionary[token]

    def size(self):
        return len(self.dictionary)

Dataset

Once the model and the tokenization process have been defined, the next step is preparing a suitable dataset to train the model. As we're aiming for simplicity to illustrate the process, let's construct a "hello world" equivalent dataset for text generation.

Our toy dataset will consist of simple, self-contained sentences. Each sentence communicates a fact about a particular animal, such as "cats rule the world" or "penguins live in the Arctic."

# Create the training data
training_data = '. '.join([
    'cats rule the world',
    'dogs are the best',
    'elephants have long trunks',
    'monkeys like bananas',
    'pandas eat bamboo',
    'tigers are dangerous',
    'zebras have stripes',
    'lions are the kings of the savannah',
    'giraffes have long necks',
    'hippos are big and scary',
    'rhinos have horns',
    'penguins live in the arctic',
    'polar bears are white'
])

With the raw data prepared, we can then tokenize it. This process converts each character in our training sentences into its corresponding token index from the vocabulary.

# Tokenize the training data
tokenized_training_data = tokenizer.tokenize(training_data)

Remember that our model expects input sequences of a fixed length, yet our sentences vary in length. To address this, we'll use padding, specifically left-padding.

Left-padding involves prepending padding tokens ("<pad>") to the beginning of shorter sequences until they match the length of the longest sequence in our dataset. By padding our sequences in this way, we ensure that every token in every sequence, no matter its original length, will be used in the training process.

# Left-pad the tokenized training data
for _ in range(max_sequence_length):
    # Prepend padding tokens
    tokenized_training_data.insert(0, tokenizer.character_to_token('<pad>'))

Input embedding

In textual data processing, one of the most critical steps is converting words or tokens into numerical representations that computational models can understand and work with. This transformation is performed by an embedding layer.

Imagine tokens as atomic units of meaning in a language - they can be words, parts of words, or even entire phrases. When tokenizing, we replace these units with corresponding integers. However, to effectively capture the nuances of language and the relationship between different tokens, we need to take a step further.

That's where the embedding layer comes in, acting as a bridge between discrete tokens and continuous vector space. It transforms each token, represented as an integer, into a continuous vector in a high-dimensional space.

Embeddings and Semantic Relationships

Let's consider a simplified example. If our tokens are words, the embedding layer helps capture the similarity between "cats" and "dogs" by positioning their corresponding vectors close together in the vector space. This closeness stems from the fact that "cats" and "dogs" often appear in similar contexts.

However, the magic doesn't stop there. As we stack multiple layers in our model, each subsequent layer can capture more and more complex concepts. The first layer might understand similar animals, like cats and dogs. Higher layers might recognize the roles these animals play in sentences, such as being pets. Even higher layers might comprehend more complex semantic structures, like cats and dogs representing different aspects of a shared concept (pets).

Creating an Embedding Layer with PyTorch

In PyTorch, the nn.Embedding class helps us create the embedding layer. During training, this layer's weights (representing our token embeddings) get updated and fine-tuned to better capture the semantic relationships between different tokens.

Here is a simplified code snippet illustrating the implementation of a token embedding layer:

class TokenEmbedding(torch.nn.Module):
    """
    PyTorch module that converts tokens into embeddings.

    Input dimension is: (batch_size, sequence_length)
    Output dimension is: (batch_size, sequence_length, d_model)
    """

    def __init__(self, d_model, number_of_tokens):
        super().__init__()
        self.embedding_layer = torch.nn.Embedding(
            num_embeddings=number_of_tokens,
            embedding_dim=d_model
        )

    def forward(self, x):
        return self.embedding_layer(x)

In the code above:

• number_of_tokens indicates the total number of unique tokens our model can encounter in the input. This number typically equals the size of our token dictionary.
• d_model specifies the size (dimensionality) of the embedding vectors. Higher dimensions allow us to encode more information about each token, but also increase the computational complexity and the time required to train the model.

The input to our TokenEmbedding module is a batch of sequences, with each token represented by an integer. The output, in contrast, is a batch of the same sequences, but each integer is now replaced by a rich, high-dimensional vector encapsulating semantic information.

Positional encoding

Why is positional encoding necessary?

Transformers operate based on the idea of self-attention, which computes relevance scores for elements in the input sequence relative to each other. However, transformers don't inherently account for the sequence order because the self-attention mechanism treats input elements independently.

In tasks like natural language understanding, the order of elements is crucially important. For instance, the sentences "The cat chases the mouse" and "The mouse chases the cat" carry very different meanings. Without considering the positional information, a transformer might interpret these sentences as equivalent. This is why we need positional encoding – to instill a sense of sequence order into transformers.

How does positional encoding work?

In the original Transformer model, positional encoding is carried out by creating a vector for each position in the sequence and adding it to the corresponding input vector. This is done so the model can learn to utilize these position vectors to understand the order of elements in the sequence.

Specifically, the positional encoding for a position "p" and dimension "i" in the input sequence is computed using these sine and cosine functions:

PE(p, 2i) = sin(p / 10000^(2i/d_model))

PE(p, 2i+1) = cos(p / 10000^(2i/d_model))

Here, "d_model" is the dimensionality of the input and output vectors in the model. A larger "d_model" would mean that each word is represented by a larger vector, allowing for more complex representations but also requiring more computational resources. The sin and cos functions are applied alternately to each dimension of the positional encoding vector, creating a pattern that the model can learn and recognize.

The resulting positional encodings vary between -1 and 1, and they have a wavelength that increases with each dimension. This pattern allows the model to distinguish different positions and to generalize to sequence lengths not encountered during training.

These positional encodings are added to the input embeddings before being processed by the transformer. This allows the transformer to learn to utilize this positional information in a way that best suits the task it's being trained on. For example, the model might learn to pay more attention to adjacent words in sentence interpretation, capturing the notion that in many languages, nearby words often have a higher syntactic and semantic relevance to each other.

The code to create a module that creates and adds such a positional embedding could be written as follows:

class PositionalEncoding(torch.nn.Module):
    """
    Pytorch module that creates a positional encoding matrix. This matrix will later be added to the 
    transformer's input embeddings to provide a sense of position of the sequence elements.
    """

    def __init__(self, d_model, max_sequence_length):
        super().__init__()
        self.d_model = d_model
        self.max_sequence_length = max_sequence_length
        self.positional_encoding = self.create_positional_encoding()

    def create_positional_encoding(self):
        """
        Creates a positional encoding matrix of size (max_sequence_length, d_model).
        """

        # Initialize positional encoding matrix
        positional_encoding = np.zeros((self.max_sequence_length, self.d_model))

        # Calculate positional encoding for each position and each dimension
        for pos in range(self.max_sequence_length):
            for i in range(0, self.d_model, 2):
                # Apply sin to even indices in the array; indices in Python start at 0 so i is even.
                positional_encoding[pos, i] = np.sin(pos / (10000 ** ((2 * i) / self.d_model)))
                
                if i + 1 < self.d_model:
                    # Apply cos to odd indices in the array; we add 1 to i because indices in Python start at 0.
                    positional_encoding[pos, i + 1] = np.cos(pos / (10000 ** ((2 * i) / self.d_model)))

        # Convert numpy array to PyTorch tensor and return it
        return torch.from_numpy(positional_encoding).float()

    def forward(self, x):
        """
        Adds the positional encoding to the input embeddings at the corresponding positions.
        """
        # Add positional encodings to input embeddings. The ":" indexing ensures we only add positional encodings up
        # to the length of the sequence in the batch. x.size(0) is the batch size, so this is a way to make sure 
        # we're not adding extra positional encodings.
        return x + self.positional_encoding[:x.size(1), :]

Masking

Remember the padding tokens we talked about before? These padding tokens don't carry any information, and it's important for the model not to pay attention to them when it's processing input data.

This is where masking comes in. A mask is typically an array that has the same length as the sentence, with ones at positions corresponding to actual words and zeros at positions corresponding to the padding tokens.

When computing attention scores, we do not want to include the effect of padding tokens. To avoid this, we can apply the mask to the attention scores matrix, effectively setting the scores at padding positions to a very large negative number (e.g., -1e9). The reason for using such a large negative number is that these scores are passed through a softmax function, which will transform these large negative numbers into zeros. This means that the padding positions will not have any influence on the final output of the attention layer.

Here is a snippet of the mask application part from the MaskedSelfAttention class:

# Apply the mask to the attention weights, by setting the masked tokens to a very low value.
# This will make the softmax output 0 for these values.
mask = mask.reshape(attention_weights.shape[0], 1, attention_weights.shape[2])
attention_weights = attention_weights.masked_fill(mask == 0, -1e9)

This line is essentially applying the mask to the attention_weights tensor. The masked_fill(mask == 0, -1e9) operation is saying "for every position in attention_weights where the corresponding position in mask is zero (i.e., a padding token), replace the attention_weights value with -1e9". This ensures that padding tokens receive no attention whatsoever.

Ideally, while predicting each new word, the decoder should only have access to the words that came before it in the sequence, not those that come after. Why? Because in real-world application, like translating a sentence, we don't have future information. The process is carried out word by word from start to end. This is known as the autoregressive property: the prediction at a certain time step depends only on the inputs and outputs at previous time steps.

To enforce this property while training the model, we use a mechanism called a causal mask (also known as a subsequent mask). This mask ensures that, for any given word, the model cannot attend to (or "peek at") the words that come after it.

Let's use an actual sentence to visualize this: "The cat sat on the mat."

If the model is trying to predict the word at the fourth position ("on"), it should only consider "The", "cat", and "sat". It should not have access to "the" or "mat". The causal mask helps us achieve this by masking out (or hiding) the future words while the model is making its prediction.

We implement this using torch.tril to create a lower-triangular matrix of ones. This ensures that position i can only attend to positions 0 through i:

# Apply causal mask to prevent attending to future positions.
# This is essential for autoregressive generation.
seq_length = x.size(1)
causal_mask = torch.tril(torch.ones(seq_length, seq_length, device=x.device)).bool()
attention_weights = attention_weights.masked_fill(~causal_mask, -1e9)

Attention

The attention mechanisms play a vital role in the transformer architecture. They're effectively a way to highlight the important parts of an input sequence, similar to how we humans pay more attention to specific parts of a sentence or image depending on the context.

Imagine we have the sentence: "The man ate a sandwich he prepared in the morning. It was topped with cheese". If we wish to understand what "It" refers to, we would naturally pay more attention to "sandwich", making it get a higher attention score. This is essentially what the attention mechanism does, calculating an attention score for each word considering all other words in the sentence.

Attention scores are derived using query, key, and value vectors. These vectors are created by multiplying the input embeddings with learned matrices (essentially a linear transformation). These vectors help calculate the attention scores, determining the impact of one token on another.

The query, key and value vectors are then used to calculate the attention scores like in the diagram below

The attention scores for the tokens are calculated as the dot product between the query and key vectors. These scores are then scaled down by dividing by the square root of the query/key/value dimension (a trick from the original Transformer paper for more stable gradients).

It's important to note that attention can be masked. For masked positions, the attention score will be set to -infinity, making these positions virtually non-existent in subsequent computations.

Finally, the attention scores are normalized using the softmax function to make them fall between 0 and 1 and sum up to 1. The normalized attention scores are then multiplied with the value vectors and summed to obtain the output of the attention layer.

In PyTorch, this is how a basic attention layer would look:

class MaskedSelfAttention(torch.nn.Module):
    """
    Pytorch module for a self attention layer.
    This layer is used in the MultiHeadedSelfAttention module.

    Input dimension is: (batch_size, sequence_length, embedding_dimension)
    Output dimension is: (batch_size, sequence_length, head_dimension)
    """

    def __init__(self, embedding_dimension, head_dimension):
        super().__init__()
        self.embedding_dimension = embedding_dimension
        self.head_dimension = head_dimension
        self.query_layer = torch.nn.Linear(embedding_dimension, self.head_dimension)
        self.key_layer = torch.nn.Linear(embedding_dimension, self.head_dimension)
        self.value_layer = torch.nn.Linear(embedding_dimension, self.head_dimension)
        self.softmax = torch.nn.Softmax(dim=-1)

    def forward(self, x, mask):
        """
        Compute the self attention.

        x dimension is: (batch_size, sequence_length, embedding_dimension)
        output dimension is: (batch_size, sequence_length, head_dimension)
        mask dimension is: (batch_size, sequence_length)

        mask values are: 0 or 1. 0 means the token is masked, 1 means the token is not masked.
        """

        # x dimensions are: (batch_size, sequence_length, embedding_dimension)
        # query, key, value dimensions are: (batch_size, sequence_length, head_dimension)
        query = self.query_layer(x)
        key = self.key_layer(x)
        value = self.value_layer(x)

        # Calculate the attention weights.
        # attention_weights dimensions are: (batch_size, sequence_length, sequence_length)
        attention_weights = torch.matmul(query, key.transpose(-2, -1))

        # Scale the attention weights.
        attention_weights = attention_weights / np.sqrt(self.head_dimension)

        # Apply the mask to the attention weights, by setting the masked tokens to a very low value.
        # This will make the softmax output 0 for these values.
        mask = mask.reshape(attention_weights.shape[0], 1, attention_weights.shape[2])
        attention_weights = attention_weights.masked_fill(mask == 0, -1e9)

        # Softmax makes sure all scores are between 0 and 1 and the sum of scores is 1.
        # attention_scores dimensions are: (batch_size, sequence_length, sequence_length)
        attention_scores = self.softmax(attention_weights)

        # The attention scores are multiplied by the value
        # Values of tokens with high attention score get highlighted because they are multiplied by a larger number,
        # and tokens with low attention score get drowned out because they are multiplied by a smaller number.
        # Output dimensions are: (batch_size, sequence_length, head_dimension)
        return torch.bmm(attention_scores, value)

Multi-head Attention for Diverse Perspectives

The attention mechanism, as we've discussed, allows a model to focus on different parts of the input sequence when generating each word in the output sequence. But there's an additional refinement we can make to this system to make it even more powerful. Instead of having one single "attention perspective", why not have multiple? That's the idea behind multi-head attention.

Imagine a class of students all looking at the same picture, but each student noticing and focusing on different details — one might notice the colors, another the shapes, another the overall composition. They each bring their own unique "perspective" to understanding the picture as a whole.

The Transformer model does something similar with multi-head attention. It has not one, but multiple sets (or "heads") of the attention mechanism, each of which independently focuses on different parts of the input. Each head can learn to pay attention to different positions of the input sequence and extract different types of information. For example, one attention head might learn to focus on syntactic aspects of the language (like grammar), while another might learn semantic information (like meaning).

Here is how it works:

Each head computes its own query, key, and value matrices from the input, just like in the single-head attention mechanism. They then compute their attention scores and produce an output. But instead of being processed individually, the output of each head is concatenated together and then transformed with a linear layer to produce the final output.

This is the code for the MaskedMultiHeadedSelfAttention class:

class MaskedMultiHeadedSelfAttention(torch.nn.Module):
    """
    Pytorch module for a multi head attention layer.

    Input dimension is: (batch_size, sequence_length, embedding_dimension)
    Output dimension is: (batch_size, sequence_length, embedding_dimension)
    """

    def __init__(self, embedding_dimension, number_of_heads):
        super().__init__()
        self.embedding_dimension = embedding_dimension
        self.head_dimension = embedding_dimension // number_of_heads
        self.number_of_heads = number_of_heads

        # Create the self attention modules
        self.self_attentions = torch.nn.ModuleList(
            [MaskedSelfAttention(embedding_dimension, self.head_dimension) for _ in range(number_of_heads)])

        # Create a linear layer to combine the outputs of the self attention modules
        self.output_layer = torch.nn.Linear(number_of_heads * self.head_dimension, embedding_dimension)

    def forward(self, x, mask):
        """
        Compute the multi head attention.

        x dimensions are: (batch_size, sequence_length, embedding_dimension)
        mask dimensions are: (batch_size, sequence_length)
        mask values are: 0 or 1. 0 means the token is masked, 1 means the token is not masked.
        """
        # Compute the self attention for each head
        # self_attention_outputs dimensions are:
        # (number_of_heads, batch_size, sequence_length, head_dimension)
        self_attention_outputs = [self_attention(x, mask) for self_attention in self.self_attentions]

        # Concatenate the self attention outputs
        # self_attention_outputs_concatenated dimensions are:
        # (batch_size, sequence_length, number_of_heads * head_dimension)
        concatenated_self_attention_outputs = torch.cat(self_attention_outputs, dim=2)

        # Apply the output layer to the concatenated self attention outputs
        # output dimensions are: (batch_size, sequence_length, embedding_dimension)
        return self.output_layer(concatenated_self_attention_outputs)

The decoder layer

The model consists of several decoder layer. Each decoder takes the output of the previous decoder as input. The first decoder takes the positional encoding layer as input. The final layer is a language model head, which is going to output the probabilies of next tokens.

In the case of an autoregressive model like GPT-2, each decoder layer is composed of a self-attention mechanism and a feed-forward neural network. The self-attention mechanism allows the model to weigh the importance of different tokens in the input when predicting the next token, while the feed-forward network enables the model to learn more traditional, positional relationships between the tokens.

When we stack multiple of these decoder layers on top of each other, it allows the model to refine its understanding of the input data over several rounds of processing. The output of one layer becomes the input of the next, and so the model can use the output of previous layers to inform its understanding of the current layer. Each layer can learn to represent different features or aspects of the input data. The higher-level layers (further in the stack) can capture more complex or abstract features, which are compositions of the simpler features captured by the lower-level layers (earlier in the stack).

So, in essence, the stacking of multiple decoder layers enables the model to capture more complex patterns and understand deeper contextual relationships among the data, thereby resulting in more accurate predictions.

So, what does each decoder layer do exactly?

Step 1: Input and Layer Normalization

When a decoder layer receives its input, the very first thing it does is apply layer normalization to these input vectors. The inputs to the decoder are high-dimensional vectors that each represent a token in the sequence.

Layer normalization is a crucial process that ensures the numerical stability of the model. It prevents numbers from getting too large during training, mitigating what's known as the "exploding gradients" problem.

But layer normalization does more than just guard against large numbers. It helps spread the data more evenly throughout the feature space, which makes the learning process more effective. By adjusting the embeddings such that each dimension has a mean of 0 and a standard deviation of 1, it ensures that no particular feature has undue influence over others due to differences in scale. This equalization of influence across dimensions is key for a balanced and efficient learning process.

Step 2: Self-Attention

Once the inputs are normalized, the decoder layer applies the self-attention mechanism. This allows the model to weigh the importance of each token in the sequence relative to the others, providing a context-aware representation of each token.

Step 3: Residual Connections

The output from the self-attention process is then added to the original input (before normalization) in what's known as a residual connection. This connection is a clever way of allowing the gradients to flow directly through the network, mitigating another common deep learning problem: vanishing gradients. The "vanishing gradients" problem refers to gradients getting exponentially smaller as they propagate back through the layers during training, making it harder for the model to learn and adjust its earlier layers.

Step 4: Another Round of Normalization and Feed-Forward Network

After applying the residual connection, the output is normalized once again and then passed through a feed-forward neural network. This network is a fully connected layer that applies non-linear transformations (like ReLU) to the data, allowing the model to learn complex relationships in the data.

Step 5: Dropout

Finally, to prevent overfitting to the training data, a dropout procedure is applied. This process randomly drops out (i.e., temporarily removes during training, 10% in my example code) a percentage of the connections in the model. In this way, it encourages the model to learn a more robust representation of the data that doesn't rely too heavily on any single connection.

After going through all these steps, the output of one decoder layer becomes the input to the next. This stacking of multiple decoders allows the model to learn to represent the data at various levels of abstraction, which is key to its ability to understand and generate human-like text.

The entire process of the decoder layer can be seen in code through the DecoderLayer and DecoderStack classes, where DecoderLayer encapsulates the processes within a single layer, and DecoderStack handles the stacking of multiple DecoderLayers.

class DecoderLayer(torch.nn.Module):
    """
    Pytorch module for an encoder layer.

    An encoder layer consists of a multi-headed self attention layer, a feed forward layer and dropout.

    Input dimension is: (batch_size, sequence_length, embedding_dimension)
    Output dimension is: (batch_size, sequence_length, embedding_dimension)
    """

    def __init__(
            self,
            embedding_dimension,
            number_of_heads,
            feed_forward_dimension,
            dropout_rate
    ):
        super().__init__()
        self.embedding_dimension = embedding_dimension
        self.number_of_heads = number_of_heads
        self.feed_forward_dimension = feed_forward_dimension
        self.dropout_rate = dropout_rate

        self.multi_headed_self_attention = MaskedMultiHeadedSelfAttention(embedding_dimension, number_of_heads)
        self.feed_forward = FeedForward(embedding_dimension, feed_forward_dimension)
        self.dropout = torch.nn.Dropout(dropout_rate)
        self.layer_normalization_1 = torch.nn.LayerNorm(embedding_dimension)
        self.layer_normalization_2 = torch.nn.LayerNorm(embedding_dimension)

    def forward(self, x, mask):
        """
        Compute the encoder layer.

        x dimensions are: (batch_size, sequence_length, embedding_dimension)
        mask dimensions are: (batch_size, sequence_length)
        mask values are: 0 or 1. 0 means the token is masked, 1 means the token is not masked.
        """

        # Layer normalization 1
        normalized_x = self.layer_normalization_1(x)

        # Multi headed self attention
        attention_output = self.multi_headed_self_attention(normalized_x, mask)

        # Residual output
        residual_output = x + attention_output

        # Layer normalization 2
        normalized_residual_output = self.layer_normalization_2(residual_output)

        # Feed forward
        feed_forward_output = self.feed_forward(normalized_residual_output)

        # Dropout, only when training.
        if self.training:
            feed_forward_output = self.dropout(feed_forward_output)

        # Residual output
        return residual_output + feed_forward_output

The DecoderStack is a number of decoder layers in sequence.

class DecoderStack(torch.nn.Module):
    """
    Pytorch module for a stack of decoders.
    """

    def __init__(
            self,
            embedding_dimension,
            number_of_layers,
            number_of_heads,
            feed_forward_dimension,
            dropout_rate,
            max_sequence_length
    ):
        super().__init__()
        self.embedding_dimension = embedding_dimension
        self.number_of_layers = number_of_layers
        self.number_of_heads = number_of_heads
        self.feed_forward_dimension = feed_forward_dimension
        self.dropout_rate = dropout_rate
        self.max_sequence_length = max_sequence_length

        # Create the encoder layers
        self.encoder_layers = torch.nn.ModuleList(
            [DecoderLayer(embedding_dimension, number_of_heads, feed_forward_dimension, dropout_rate) for _ in
             range(number_of_layers)])

    def forward(self, x, mask):
        decoder_outputs = x
        for decoder_layer in self.encoder_layers:
            decoder_outputs = decoder_layer(decoder_outputs, mask)

        return decoder_outputs

The feed forward layer

class FeedForward(torch.nn.Module):
    """
    Pytorch module for a feed forward layer.

    A feed forward layer is a fully connected layer with a ReLU activation function in between.
    """

    def __init__(self, embedding_dimension, feed_forward_dimension):
        super().__init__()
        self.embedding_dimension = embedding_dimension
        self.feed_forward_dimension = feed_forward_dimension
        self.linear_1 = torch.nn.Linear(embedding_dimension, feed_forward_dimension)
        self.linear_2 = torch.nn.Linear(feed_forward_dimension, embedding_dimension)

    def forward(self, x):
        """
        Compute the feed forward layer.
        """
        return self.linear_2(torch.relu(self.linear_1(x)))

The model

the LanguageModel class brings together all the layers we've discussed so far: token embeddings, positional encoding, normalization, and the decoder stack. Let's go through each component:

• Token Embedding: The model first creates token embeddings. Each token (word, subword, or character) in the input sequence is represented by a high-dimensional vector, which is initialized randomly and learned during training. The embedding dimension is a hyperparameter.
• Positional Encoding: The model then applies positional encoding to the token embeddings. This helps the model to understand the order of the tokens in the sequence, a crucial piece of information in many natural language tasks. In GPT, positional encoding is achieved by adding a vector to each token embedding, which encodes the token's position in the sequence.
• Layer Normalization: Next, the model applies layer normalization to the result of the positional encoding. This normalization process adjusts the vectors so that each dimension has a mean of 0 and a standard deviation of 1, equalizing the influence of each dimension and helping to stabilize the learning process.
• Decoder Stack: The normalized output is then passed into a stack of decoders. Each decoder block in the stack consists of a self-attention mechanism and a feed-forward neural network, with layer normalization and residual connections used throughout.

After the data passes through the decoder stack, we reach the final part of the model: the language model head.

class LanguageModel(torch.nn.Module):
    """
    Pytorch module for a language model.
    """

    def __init__(
            self,
            number_of_tokens,  # The number of tokens in the vocabulary
            max_sequence_length=512,  # The maximum sequence length to use for attention
            embedding_dimension=512,  # The dimension of the token embeddings
            number_of_layers=6,  # The number of decoder layers to use
            number_of_heads=4,  # The number of attention heads to use
            feed_forward_dimension=None,  # The dimension of the feed forward layer
            dropout_rate=0.1  # The dropout rate to use
    ):
        super().__init__()
        self.number_of_tokens = number_of_tokens
        self.max_sequence_length = max_sequence_length
        self.embedding_dimension = embedding_dimension
        self.number_of_layers = number_of_layers
        self.number_of_heads = number_of_heads

        if feed_forward_dimension is None:
            # GPT-2 paper uses 4 * embedding_dimension for the feed forward dimension
            self.feed_forward_dimension = embedding_dimension * 4
        else:
            self.feed_forward_dimension = feed_forward_dimension

        self.dropout_rate = dropout_rate

        # Create the token embedding layer
        self.token_embedding = TokenEmbedding(embedding_dimension, number_of_tokens)

        # Create the positional encoding layer
        self.positional_encoding = PositionalEncoding(embedding_dimension, max_sequence_length)

        # Create the normalization layer
        self.layer_normalization = torch.nn.LayerNorm(embedding_dimension)

        # Create the decoder stack
        self.decoder = DecoderStack(
            embedding_dimension=embedding_dimension,
            number_of_layers=number_of_layers,
            number_of_heads=number_of_heads,
            feed_forward_dimension=self.feed_forward_dimension,
            dropout_rate=dropout_rate,
            max_sequence_length=max_sequence_length
        )

        # Create the language model head
        self.lm_head = LMHead(embedding_dimension, number_of_tokens)

    def forward(self, x, mask):
        # Compute the token embeddings
        # token_embeddings dimensions are: (batch_size, sequence_length, embedding_dimension)
        token_embeddings = self.token_embedding(x)

        # Compute the positional encoding
        # positional_encoding dimensions are: (batch_size, sequence_length, embedding_dimension)
        positional_encoding = self.positional_encoding(token_embeddings)

        # Post embedding layer normalization
        positional_encoding_normalized = self.layer_normalization(positional_encoding)

        decoder_outputs = self.decoder(positional_encoding_normalized, mask)
        lm_head_outputs = self.lm_head(decoder_outputs)

        return lm_head_outputs

Language Model Head

The LanguageModel class concludes with the LMHead layer. This layer is essentially a linear transformation that maps the high-dimensional output of the decoder stack back down to the dimension of the token vocabulary.

To put it in simpler terms, imagine you have a vocabulary of 10,000 unique words and your decoder's output dimension is 512. Each word in your vocabulary is going to have a unique 512-dimensional vector representation after passing through the decoder. But we want to assign a probability to each word in the vocabulary given the preceding context. This is where LMHead comes in.

LMHead maps the 512-dimensional vector back to the 10,000-dimensional space, essentially assigning a score to each of the 10,000 words. These scores are then passed through a softmax function to convert them into probabilities. Therefore, LMHead is crucial in transforming the high-dimensional output of the decoder stack to the likelihood of each word being the next word in the sequence.

The LMHead is implemented as a subclass of PyTorch's torch.nn.Module. It uses a linear layer (a basic type of neural network layer that applies a linear transformation to the input) to map the decoder's output dimension to the number of tokens in the vocabulary.

Consider a scenario where the model has processed the phrase "The quick brown fox". The decoder stack outputs a 512-dimensional vector for the final word "fox".

Now, let's imagine our vocabulary consists of just five words: "jumps", "sleeps", "eats", "runs", and "fox". We want to determine the likelihood of each of these words being the next word in the sentence.

That's where the LMHead comes into play. This linear layer maps the 512-dimensional vector to a 5-dimensional vector, since we have 5 words in our vocabulary. Each entry in this output vector corresponds to a word in the vocabulary, and can be interpreted as a raw score for how likely that word is to follow the phrase "The quick brown fox".

Assume the probabilities after applying softmax are: ["jumps": 0.6, "sleeps": 0.05, "eats": 0.2, "runs": 0.13, "fox": 0.02]. So the model predicts "The quick brown fox jumps" as the most likely continuation of the input sequence.

class LMHead(torch.nn.Module):
    """
    Pytorch module for the language model head.
    The language model head is a linear layer that maps the embedding dimension to the vocabulary size.
    """

    def __init__(self, embedding_dimension, number_of_tokens):
        super().__init__()
        self.embedding_dimension = embedding_dimension
        self.number_of_tokens = number_of_tokens
        self.linear = torch.nn.Linear(embedding_dimension, number_of_tokens)

    def forward(self, x):
        """
        Compute the language model head.

        x dimensions are: (batch_size, sequence_length, embedding_dimension)
        output dimensions are: (batch_size, sequence_length, number_of_tokens)
        """
        # Compute the linear layer
        # linear_output dimensions are: (batch_size, sequence_length, number_of_tokens)
        linear_output = self.linear(x)

        return linear_output

Autoregressive Wrapper

To allow our language model to generate text one token at a time, we wrap it in an autoregressive module. In an autoregressive model, the output from previous steps is fed as input to subsequent steps. This is where the AutoregressiveWrapper comes in.

The autoregressive wrapper takes a sequence of tokens as input, where the sequence length is one token more than the maximum sequence length allowed. This extra token is necessary because in autoregressive models, the model is trained to predict the next token in the sequence given the previous tokens.

For instance, consider the sentence: "The cat sat on the mat". If the input sequence is "The cat sat on the", the target output sequence (the sequence that the model is trained to predict) is "cat sat on the mat". This is why the input sequence is shifted one step to the left relative to the output sequence.

The AutoregressiveWrapper class also includes a method for calculating the probabilities of the next token in the sequence. It generates these probabilities by applying a softmax function to the logits (the raw, unnormalized scores output by the model for each token in the vocabulary) associated with the last token in the sequence. A "temperature" parameter is used to adjust the sharpness of the probability distribution. Lower temperatures make the output more deterministic (i.e., more likely to choose the most probable token), while higher temperatures make the output more random.

class AutoregressiveWrapper(torch.nn.Module):
    """
    Pytorch module that wraps a GPT model and makes it autoregressive.
    """

    def __init__(self, gpt_model):
        super().__init__()
        self.model = gpt_model
        self.max_sequence_length = self.model.max_sequence_length

    def forward(self, x, mask):
        """
        Autoregressive forward pass
        """
        inp, target = x[:, :-1], x[:, 1:]
        mask = mask[:, :-1]

        output = self.model(inp, mask)
        return output, target

    def next_token_probabilities(self, x, mask, temperature=1.0):
        """
        Calculate the token probabilities for the next token in the sequence.
        """
        logits = self.model(x, mask)[:, -1]

        # Apply the temperature
        if temperature != 1.0:
            logits = logits / temperature

        # Apply the softmax
        probabilities = torch.softmax(logits, dim=-1)

        return probabilities

Trainer

Now we have a model that can learn how language works we need to actually train it to do so.

First we create the tokenizer, so we can convert our dataset to tokens.
Then we create the autoregressive language model. Because the dataset is very small I am going to only set a max_sequence_length of 20, but a more normal value would be 512.

I will then split the training text into sequences of [max_sequence_length + 1]. To make sure the starting tokens are also considered the data will be padded on the left.

Then we are going to train the model for 100 epochs, with a batch size of 8. An epoch means a complete pass over the training data. Batch size means that in every forward pass through the model we consider 8 sequences from the training data simultaneously. The higher the batch size the better the model can learn patterns in the data, but a higher batch size also leads to more memory usage.

def create_training_sequences(max_sequence_length, tokenized_training_data):
    # Create sequences of length max_sequence_length + 1
    # The last token of each sequence is the target token
    sequences = []
    for i in range(0, len(tokenized_training_data) - max_sequence_length - 1):
        sequences.append(tokenized_training_data[i: i + max_sequence_length + 1])
    return sequences


def tokenize_and_pad_training_data(max_sequence_length, tokenizer, training_data):
    # Tokenize the training data
    tokenized_training_data = tokenizer.tokenize(training_data)
    for _ in range(max_sequence_length):
        # Prepend padding tokens
        tokenized_training_data.insert(0, tokenizer.character_to_token('<pad>'))
    return tokenized_training_data


tokenizer = Tokenizer()

embedding_dimension = 256
max_sequence_length = 20
number_of_tokens = tokenizer.size()

# Create the model
model = AutoregressiveWrapper(LanguageModel(
    embedding_dimension=embedding_dimension,
    number_of_tokens=number_of_tokens,
    number_of_heads=4,
    number_of_layers=3,
    dropout_rate=0.1,
    max_sequence_length=max_sequence_length
))

# Create the training data
training_data = '. '.join([
    'cats rule the world',
    'dogs are the best',
    'elephants have long trunks',
    'monkeys like bananas',
    'pandas eat bamboo',
    'tigers are dangerous',
    'zebras have stripes',
    'lions are the kings of the savannah',
    'giraffes have long necks',
    'hippos are big and scary',
    'rhinos have horns',
    'penguins live in the arctic',
    'polar bears are white'
])

tokenized_and_padded_training_data = tokenize_and_pad_training_data(max_sequence_length, tokenizer, training_data)
sequences = create_training_sequences(max_sequence_length, tokenized_and_padded_training_data)

# Train the model
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
trainer = Trainer(model, tokenizer, optimizer)
trainer.train(sequences, epochs=100, batch_size=8)

The trainer is a helper class that loops over the epochs and shuffles the data at the start of each epoch. The reason to do this is to prevent the batches from being the same every time, causing the model to overfit to these specific batches.

While creating batches we also determine the mask. All padding tokens are masked, meaning they will not be considered in the attention step.

Then we do a forward pass through the model with a batch. This means we let the model make predictions using the given data. The predictions are then compared to a target value, which is the sequence shifted by 1 step so the next token becomes visible. The model outputs probabilities for what token should be the next. The loss function knows what the answer should be. The further from its target the prediction was the higher the loss value will be.

When the loss value is calculated the model can be updated. This is done by calculating gradients; the direction the weights should be adjusted to improve the prediction of the model. The model is then slightly adjusted in the direction of the gradients, and a new batch can be processed.

If everything works as planned, the loss should go down over time. I return the loss per epoch, so it can be plotted.

class Trainer:

    def __init__(self, model, tokenizer: Tokenizer, optimizer=None):
        super().__init__()
        self.model = model
        if optimizer is None:
            self.optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
        else:
            self.optimizer = optimizer
        self.tokenizer = tokenizer
        self.loss_function = torch.nn.CrossEntropyLoss()

    def train(self, data: List[str], epochs, batch_size):
        loss_per_epoch = []
        for epoch in range(epochs):
            losses = []

            # Shuffle the sequences
            random.shuffle(data)

            # Create batches of sequences and their respective mask.
            batches = []
            for i in range(0, len(data), batch_size):
                sequence_tensor = torch.tensor(data[i: i + batch_size], dtype=torch.long)

                # Create the mask tensor for the batch, where 1 means the token is not a padding token
                mask_tensor = torch.ones_like(sequence_tensor)
                mask_tensor[sequence_tensor == self.tokenizer.character_to_token('<pad>')] = 0

                batches.append((sequence_tensor, mask_tensor))

            # Train the model on each batch
            for batch in batches:
                self.model.train()

                # Create the input and mask tensors
                input_tensor = torch.zeros((batch_size, self.model.max_sequence_length + 1), dtype=torch.long)
                mask_tensor = torch.zeros((batch_size, self.model.max_sequence_length + 1), dtype=torch.long)

                for i, input_entry in enumerate(batch[0]):
                    input_tensor[i] = input_entry

                for i, mask_entry in enumerate(batch[1]):
                    mask_tensor[i] = mask_entry

                # Compute the model output
                model_output, target = self.model.forward(x=input_tensor, mask=mask_tensor)

                # Compute the losses
                # The loss is computed on the model output and the target
                loss = self.loss_function(model_output.transpose(1, 2), target)

                # Backpropagate the loss.
                loss.backward()

                # Clip the gradients. This is used to prevent exploding gradients.
                torch.nn.utils.clip_grad_norm_(self.model.parameters(), 0.5)

                # Update the model parameters. This is done by taking a step in the direction of the gradient.
                self.optimizer.step()

                # Reset the gradients. This is done so that the gradients from the previous batch
                # are not used in the next step.
                self.optimizer.zero_grad()

                # Append the loss to the list of losses, so that the average loss can be computed for this epoch.
                losses.append(loss.item())

            # Print the loss
            epoch_loss = np.average(losses)
            loss_per_epoch.append(epoch_loss)
            print('Epoch:', epoch, 'Loss:', epoch_loss)

        return loss_per_epoch

Evenutally plotting the loss should give us a nice graph with decreasing loss. It is plotted in log scale, so you can see the smaller variations towards the end of training.

# Plot the loss per epoch in log scale
plt.plot(loss_per_epoch)
plt.yscale('log')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.show()

Generator

Now the model is trained and I want to see if it actually learned to write. Let's try to generate a text based on the prompt "elephants". I want it to continue writing for 50 tokens.  

Since the only mention of elephants in the training data is "elephants have long trunks", I expect the model to write this.

max_tokens_to_generate = 50
generator = Generator(model, tokenizer)
generated_text = generator.generate(
    max_tokens_to_generate=max_tokens_to_generate,
    prompt="elephants",
    padding_token=tokenizer.character_to_token('<pad>')
)
print(generated_text.replace('<pad>', ''))

but first we need to write the code for the Generator. A helper class for generating text.

First we switch the model from 'training' mode to 'eval' mode. In eval mode the model will not apply dropout. We also wrap the generation in torch.no_grad() to disable gradient computation, which saves memory during inference.

The prompt we give is converted to tokens, and then padded so it has the correct sequence length.

Then we are going to auto-regressively generate new tokens and add them to the input sequence. After a token is added we run the new input sequence with the extra token through the model again, and we append a new token. We continue this process until the maximum number of characters we wanted to generate is reached, or until we have generated the eos_token, or end of sequence token. This is a token that can be defined by the user as an indication that we need to stop generating.

def pad_left(sequence, final_length, padding_token):
    return [padding_token] * (final_length - len(sequence)) + sequence


class Generator:

    def __init__(
            self,
            model,
            tokenizer):
        self.model = model
        self.tokenizer = tokenizer

    def generate(
            self,
            max_tokens_to_generate: int,
            prompt: str = None,
            temperature: float = 1.0,
            eos_token: int = None,
            padding_token: int = 0):

        self.model.eval()

        if prompt is None:
            start_tokens = [self.tokenizer.character_to_token(padding_token)]
        else:
            start_tokens = self.tokenizer.tokenize(prompt)

        input_tensor = torch.tensor(
            pad_left(
                sequence=start_tokens,
                final_length=self.model.max_sequence_length + 1,
                padding_token=padding_token
            ),
            dtype=torch.long
        )

        num_dims = len(input_tensor.shape)

        if num_dims == 1:
            input_tensor = input_tensor[None, :]

        out = input_tensor
        for _ in range(max_tokens_to_generate):

            x = out[:, -self.model.max_sequence_length:]

            mask = torch.ones_like(x)
            mask[x == padding_token] = 0

            # Compute the next token probabilities
            next_token_probabilities = self.model.next_token_probabilities(
                x=x,
                temperature=temperature,
                mask=mask
            )

            # Sample the next token from the probability distribution
            next_token = torch.multinomial(next_token_probabilities, num_samples=1)

            # Append the next token to the output
            out = torch.cat([out, next_token], dim=1)

            # If the end of sequence token is reached, stop generating tokens
            if eos_token is not None and next_token == eos_token:
                break

        generated_tokens = out[0].tolist()
        return ''.join([self.tokenizer.token_to_character(token) for token in generated_tokens])

The training is finished and the generator has run. aaaaaanndd... the model actually outputs the text we trained it on!

Of course, when training on such a small dataset the model will completely overfit it and learn to reproduce the whole dataset. Not what you are usually looking for in a language model, but in this "hello world" test it means success!

Saving and loading the model

Once you trained the model, it is useful if you can save it, so you don't have to train a new model every time.

To do this we add the following code to the LanguageModel class.

def save_checkpoint(self, path):
    print(f'Saving checkpoint {path}')
    torch.save({
        'number_of_tokens': self.number_of_tokens,
        'max_sequence_length': self.max_sequence_length,
        'embedding_dimension': self.embedding_dimension,
        'number_of_layers': self.number_of_layers,
        'number_of_heads': self.number_of_heads,
        'feed_forward_dimension': self.feed_forward_dimension,
        'dropout_rate': self.dropout_rate,
        'model_state_dict': self.state_dict()
    }, path)

@staticmethod
def load_checkpoint(path) -> 'LanguageModel':
    checkpoint = torch.load(path)
    model = LanguageModel(
        number_of_tokens=checkpoint['number_of_tokens'],
        max_sequence_length=checkpoint['max_sequence_length'],
        embedding_dimension=checkpoint['embedding_dimension'],
        number_of_layers=checkpoint['number_of_layers'],
        number_of_heads=checkpoint['number_of_heads'],
        feed_forward_dimension=checkpoint['feed_forward_dimension'],
        dropout_rate=checkpoint['dropout_rate']
    )
    model.load_state_dict(checkpoint['model_state_dict'])
    return model

Since we use the AutoregressiveWrapper as convenience class, we can give this wrapper the save and load methods too.

def save_checkpoint(self, path):
    self.model.save_checkpoint(path)

@staticmethod
def load_checkpoint(path) -> 'AutoregressiveWrapper':
    model = LanguageModel.load_checkpoint(path)
    return AutoregressiveWrapper(model)

This makes it possible to easily save and load a trained model using.

model.save_checkpoint('./trained_model')
model = model.load_checkpoint('./trained_model')

Running on GPU

If you have a GPU at your disposal and CUDA is configured, you can use a GPU to speed up training.

First we need to define a function to determine if we can use a GPU:

def get_device():
    if torch.cuda.is_available():
        return torch.device('cuda')
    else:
        return torch.device('cpu')

Then we need to move the model and all input tensors to our device. We can do this by using the torch function to.

When creating the model, move it to the device:

model = AutoregressiveWrapper(LanguageModel(
    embedding_dimension=embedding_dimension,
    number_of_tokens=number_of_tokens,
    number_of_heads=4,
    number_of_layers=3,
    dropout_rate=0.1,
    max_sequence_length=max_sequence_length
)).to(get_device())

In the Trainer.train method, move the input tensors and target to the device:

model_output, target = self.model.forward(
    x=input_tensor.to(get_device()),
    mask=mask_tensor.to(get_device())
)

loss = self.loss_function(model_output.transpose(1, 2), target.to(get_device()))

In the Generator.generate method, move the input tensor to the device:

input_tensor = torch.tensor(
    pad_left(
        sequence=start_tokens,
        final_length=self.model.max_sequence_length + 1,
        padding_token=padding_token
    ),
    dtype=torch.long
).to(get_device())

Note that we also use register_buffer for the positional encoding matrix in the PositionalEncoding class. This ensures that the precomputed positional encodings automatically move to the correct device along with the model, avoiding device mismatch errors.

The speedup achieved by using a GPU can be really significant.

Conclusion

While writing this blogpost I learned a lot about how the transformer works, and how we can use it to generate text. I hope you enjoyed reading it, and learned something new too!

Feel free to leave a comment if you have remarks, questions or just want to let me know what you think :)

The code for this blogpost is available at https://github.com/wingedsheep/transformer

Sources

Some resources that were incredibly helpful in helping me understand attention, the transformer architecture and GPT models.

• Attention is all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin https://arxiv.org/abs/1706.03762
• Transformer implementation by Phil Wang (Lucidrains) https://github.com/lucidrains/x-transformers
• The illustrated transformer by Jay Alammar https://jalammar.github.io/illustrated-transformer/
• Illustrated GPT-2 by Jay Alammar https://jalammar.github.io/illustrated-gpt2/
• MinGPT by Andrej Karpathy https://github.com/karpathy/minGPT
• Seq2seq and attention by Lena Voita https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html
• The GPT-3 Architecture, on a Napkin by Daniel Dugas https://dugas.ch/artificial_curiosity/GPT_architecture.html

Introduction

Since the release of stable diffusion there has been an avalanche of new techniques and tools to create AI generated art. One of the most interesting of these tools is dreambooth. This is a technique created by Google Research to personalize text to image models.

A text to image model like stable diffusion allows you to create images from a description. For example "A beautiful oil painting of a cat sailing the ocean" will probably give you something like the image below with the default stable diffusion model.

These models provide the relations between how the world looks and how it is described in text in a very compact format. If you know exactly the right text you can get the model to output almost any image you want. But the catch is that it often takes a large effort and a lot of experimenting to find the right text to use as input.

This is where Dreambooth comes in. Dreambooth allows you to teach the model your own c0ncepts. In The example below they trained the model the concept of a certain cute dog with large ears. This dog can then be placed in different contexts.

Dreambooth unlocks many possibilities. You could train a model the concept of you by feeding it selfies and place yourself in your favourite movies, you could turn your pets into Pixar characters or teach it to recognize a certain art style that you like.

Training a custom Dreambooth model

This part will be a guide for training your own Dreambooth model using the colab notebook found in https://github.com/TheLastBen/fast-stable-diffusion

I am going to use my cat Pip as an example.

Data

First we need some data. In the Github repository they recommend collecting 200 or more pictures as training data, but good results can be achieved with a lot less.

As preparation I collected 39 pictures of my cat. All pictures need to have the dimensions of 512 x 512, because that is the format stable diffusion is trained on. I use https://pixlr.com/ to resize and crop the images.

All the resized images go into a folder in my Google Drive.

After all the images are prepared and uploaded we can start the colab notebook. Make sure that the runtime type is set to GPU.

Training the model

Mount google drive

The first step is to mount your Google Drive to Colab. The notebook saves the trained model to your drive. You can also use a folder from your drive for reference images used to train the model.

Setup environment & download the model

After mounting Google Drive we can setup the environment and enter our Huggingface token. Huggingface hosts the stable diffusion model here https://huggingface.co/CompVis/stable-diffusion-v1-4. You can request access to the model if you agree to some terms and conditions. To do this you need to create an account.

Dreambooth settings

Next we have to configure Dreambooth for the model we are going to train. There are some fields we have to fill in.

SUBJECT_NAME is the class of the object we are training on. This is a  coarse description of the category our training subject belongs to. It could be something like "person", "car", "house" etc. In my case I can enter "cat" here, but if you want to train the model on your own face for example you might enter "face". Dreambooth uses this to entangle your instance with the subject. The result is that less training time is needed, because the base model already knows a lot about your subject if it is general enough. Stable diffusion has seen all the poses cats can take.

INSTANCE_NAME is how we are going to refer to the subject. It is good to have a unique description here, that does not have a lot of associations with it. If we pick a word that already has a lot of associations the finetuning process has to override all those associations. I chose "pipper2022", but maybe a completely random string like "2Np1Pu6EB3" would work even better.

INSTANCE_DIR_OPTIONAL here I am going to refer to my Google Drive folder containing all the images of my cat Pip. It is not necessary to refer to a folder. If you leave this field empty you can upload the files directly in colab.

Number_of_subject_images here you can enter the number of subject images. These are images of your general class, so in my case images of cats. If you don't want to supply your own images they will be generated using stable diffusion, by using SUBJECT_NAME as a prompt. Some prompts generate low quality images, so if you want better quality you can choose to upload your own class images.

Subject images

For the next step we can indicate a folder with subject images. These are images containing the category of our training subject. In my case these should be images of random cats. For this blogpost I chose not to supply my own cat images, since stable diffusion did quite a good job generating them for me.

Below are some examples of the cats generated by the notebook using the prompt "cat".

Training the model

Then it is time to train the model. The author of the Colab notebook recommends 2000 steps. How many steps you need really depends a lot on the complexity of the subject it seems. I found that for my cat 1000 steps and 40 images is sufficient. Some people get great results with few training samples and steps, but if you want to capture a person in different angles and distances you would probably need more.

Roope Rainisto did a great series of images of Grace Kelly, where he used 10.000 training steps according to one of the follow up tweets. But the drawback might be overfitting that leads to every person looking like your subject.

If you didn't supply subject images these will be generated before training the model. Generating the images and training the model will take time, so you can get a cup of coffee while you wait for the results.

The generated class images are saved to the data folder in your "content/data" folder, so you can check if the generated images represent your class well.

Because this is a test model I set the number of training steps to 1000 and let it run. Generating the 200 class images takes a bit more than 20 minutes and training the model for 1000 steps takes around 25 minutes on Colab Pro.

Update 2022-10-30: TheLastBen dreambooth implementation is updated. You only need to upload around 30 images, and no class images are needed anymore. This greatly reduces the amount of preparation required. Recommended steps is 100x the number of training images uploaded.

Generating images

Now we trained the model we should be able to use it to transport my cat Pip to various settings. Running the last segment will start a local server for stable diffusion web-ui. For me the local server didn't work, so I used the Gradio server.

We can now open the web-ui and use the SUBJECT_NAME (cat) and the INSTANCE_NAME (pipper2022) to generate images. In the example below I tried to make an oil painting of Pip, using the name of a made up artist called "Arthur Valin".

Here are some of the best Pips to come out of the process.

A collection of fantastic beasts and magical creatures in their natural environments. The descriptions of the creatures are generated using GPT-3 by giving it a list of imaginary creatures that I made up. The results are then given to Dall-e 2, to see how creative it is in turning imaginary creature names into pictures.

My part in this process was creating an initial list of imaginary creatures to use as examples f0r GPT-3, cherry picking the best pictures, and tweaking the creature descriptions where necessary.

Enjoy!

Klouzet

The Klouzet is a creature that lives in forests and near rivers. It is a carnivore that hunts for fish, and uses its long tongue to groom its fur. It is the size of a large dog, and has bright green eyes. It has long pointy ears, and a thick fur that protects it from the cold. The fur is very rough, and has green stripes from front to back for better camouflage in the forest. In the early morning you can often find them near rivers, hunting for fish. It is a timid creature, and will flee if it senses danger. The Klouzet is an important creature in the forest ecosystem, as it helps to keep the fish population in check. Without the Klouzet, the fish would over populate and eventually deplete the resources in the river. During the winter, the Klouzet will hibernate in its burrow, and emerge in the spring when the fish begin to spawn.

Flotop

The Flotop is a creature that lives in hot and dry desert areas. It feeds on termites. It is the size of a small bird, and has a pink body and a long greenish tail. It has a pointy snout, like a bird's beak. The Flotop looks scary, but it is mostly harmless to humans. It is a very efficient flyer, because of its ultra lightweight body. It can fly for long periods of time without tiring. Its long tail can be used to get termites out of their mounds. Interestingly, the Flotop does not have any eyes. It uses its sense of smell to locate termite hills.

Bortodont

The Bortodont is a creature that lives in swamps. It is a slow moving creature that feeds on nuts, worms, and other insects. It is the size of an Ankylosaurus, and has a grayish body with large pointy brownish extensions coming out of its side. The extensions are flexible and can be used to help the Bortodont move through the mud. The Bortodont's back is very tough, and it has armour to protect it from predators. When it is threatened, it will raise its extensions to make itself look bigger and more intimidating.

Flisk

The Flisk is a creature that lives in underwater kelp forests. It looks like a seal with large mouse like ears. These ears serve as sonar, to help the Flisk find food. It is the size of a small seal, and has white fur with large mouse-like ears. It is a good swimmer and can stay under water for a long time.  Flisks are very curious, and will even approach humans if it feels safe. A Flisk's diet consists mostly of kelp, but it will also eat small fish and invertebrates. Larger Flisks can even eat octopuses and squid.

Grimalkin

The Grimalkin is a predatory cat that lives in the coldest parts of the world. It has black fur and green eyes. It is about a meter long, and has long claws that help it to stalk and kill its prey. The Grimalkin is a very stealthy hunter, and will often lie in wait for hours before attacking. Its favorite prey is the hare, but it will also eat other small mammals and birds. In the winter, the Grimalkin will often hunt in the snow, using its black fur to camouflage itself as a rock. Some people believe that the Grimalkin is a magical creature, because of its ability to disappear and reappear at will.

Gronack

The Gronack is a gentle giant that lives in forests. It is very tall and has a long neck. It eats the leaves from the trees. To avoid being seen, they often stand very still amongst the trees. When they move, they are very graceful. Gronacks are very social creatures, and live in herds. Interestingly, Gronacks are born with stripes, which fade as they grow older.

Draug

The Draug is a bird that lives near the sea. It is a large raven like creature with black wings. It has red eyes, and can grow up to one and a half meters tall, with a wingspan of 3,5 meters. The Draug is a scavenger, and will eat anything it can find. It is also known to attack and kill small animals. The Draug is a feared creature, because of its size and its red eyes. Some people believe that the Draug is a demonic creature, because of its appearance.

Pryderi

The Pryderi is a small mammal that lives in fields and grasslands. It is small and fast, and has bright red fur. It looks a bit like a small coyote, and has orange-brown fur with black spots, and large ears. It has very intense orange eyes. It feed most mostly on insects, mice and eggs. It is known for chasing butterflies. Pryderis grow up to 50 cm long, and are known to stay together with their siblings for their whole lives.

Quillbeast

The quillbeast is a large creature that swims in oceans. It has a long body and tentacles. It is transparent, and eats plankton. It can swim very fast, and is often seen near tropical islands. They are harmless to humans, and are actually quite friendly. They often play by swimming in circles around each other.

Brollachan

The Brollachan is a predatory creature that lurks in the shadows of dark forests. It has green eyes and long, sharp claws. It is muscular and stocky, with brown and grey stripes. The Brollachan is a cunning hunter. Its favorite prey is deer, but it will also eat other small mammals and birds. It is a feared creature, because of its size and its sharp claws. Sometimes the Brollachan will even attack humans.

Limnade

The limnade is a creature that lives in freshwater ponds and lakes. It has a long neck and an elegant body. The limnade is pink like a flamingo, but has a thin long beak, making it look very elegant. It feeds on algae and small fish. Limnades keep the water clear by eating algae.

Sundor

The Sundor is a nocturnal creature that hunts at night. It has bright red fur, and a long tail. It is the size of a fox, and has black stripes. Sundors are very shy creatures, and will only come out at night. During the daytime, they will sleep in their dens or bask in the sun. Sundors are skilled hunters, and will often hunt small rodents They are also known to eat fruits on occasion. To avoid being followed, they will often use their long tail to cover their tracks. Sun reflecting on their fur sometimes makes it look like they are on fire.

Orox

The Orox is a small mammal that lives in fields and grasslands. It is shy and will run away if it sees you. It is the size of a small goat, and has a pointy beak, small straight horns, and large ears. It is colored like dry grass for camouflage, and has black stripes. The Orox is a herbivore, and feeds on leaves and grass. It is a very fast runner, and can reach speeds of up to 60 km/h. When it is threatened, it will often use its speed to escape.

Snorble

The Snorble is a small mammal that lives in meadows and forests. It is very fluffy, and has a long snout. It is covered in short fur that can appear to have multiple colours. The Snorble is a timid creature, and will often flee if it senses danger. It is a good jumper, and can leap up to 2 meters high. The Snorble is a herbivore, and feeds on leaves, flowers and fruits. It is also known to eat insects on occasion. Snorbles are very social creatures, and live in large families. They are known for their friendly personalities and their love of napping in sunbeams.

Cranat

The Cranat is a bird that lives near rivers and wetlands. It has red and orange feathers. It is the size of a crane, and has long wings that help it to fly. The Cranat is a skilled hunter, and will catch insects in its long beak. It is also known to eat small fish and amphibians. Some people believe that the Cranat is a sacred creature, because of its beautiful feathers.

Trog

The Trog is a large, muscular creature that lives in dark caves. It has a very good sense of smell and can find its way in the dark. Trogs are very strong and can lift heavy rocks. They are also very fast and agile. Trogs are omnivores, and will eat anything they can find. Their favorite food is cave fish, but they will also eat insects, rodents, and even other Trogs. They are known to be very territorial, and will fight to the death to defend their cave.

Mosslits

Mosslits are small snail like creatures that live in forests. They light up in the dark so they can help each other to find food. Their long tails have a light sensing organ at the end, which helps them to find their way around. Mosslits are herbivores, and their favorite food is leaves. They will also eat other plant matter, such as fruits and flowers.

Jebble

The Jebble is a creature that lives on mountain tops. It is the same color as the sky, so it is very hard to see when it is not moving. It is the size of a large rabbit, and has blue-gray fur. The Jebble is a herbivore, and feeds on grass and other plants. Jebbles will often roll in the snow to cool down.

Glim

The Glim is a small creature that lives in cities. It is attracted to light, and can often be seen running around at night. It has brightly coloured fur, and is the size of a rat. Glims are known for rummaging through trash cans in search of food. They are also known to eat insects. Glims are not harmful to humans, but they can be a nuisance.

Direhorn

Direhorns are very large and dangerous creatures that live in the mountains. They have two long horns and sharp teeth. They are the size of a large bull, and can weigh up to 1,000 kg. Direhorns are very aggressive, and will often charge at anything that they perceive as a threat. They often attack and kill smaller animals but will also eat plants. Direhorns are a danger to humans, and should be avoided if possible.

Toximander

The Toximander is a lizard that lives in the jungle. It is brightly colored with green and yellow stripes. It is a good climber, and often eats fruit high up in the trees. The name of the toximander comes from the Latin word "toxicus", meaning poisoned. This is because the toximander is poisonous to touch. If you see one, it is best to leave it alone.

Qu'lith

The qu'lith is a predatory creature that lives in the desert. It has brown and grey fur, and a forked tongue. It is the size of a small cat, and has two thin, pointy ears. The qu'lith is a fast creature, and has sharp claws. It will often lie in wait for prey, and then pounce on it. Its favorite prey is lizards, but it will also eat small mammals and birds. It can survive without water for long periods of time, because it gets moisture from the prey it eats.

Glonk

The Glonk is a creature that lives in jungles. It is very large, and its skin is tough and scaly. It has a long tail and a short snout. It is a herbivore, and feeds on fruit. Glonks are very shy creatures, and will often stay in the trees to avoid being seen. If they are threatened, they will use their sharp claws to defend themselves. Ever since they were discovered, Glonks have been hunted for their skin, which is used to make leather. Only small populations of Glonks can still be found, hidden far in the jungle.

Yellow lanternfish

The yellow lantern fish is a small fish that lives in the deep ocean. Unlike its scary cousin that lures fish with its light, this variant is peaceful and uses its light to communicate with other lantern fish. It can also produce different light patterns to camouflage itself from predators. It is a yellow fish with a light on its chin, and has large eyes. It is related to the angler fish. The lantern fish is a very social creature, and often congregates in large groups.

Treestrider

The Treestrider is a creature that lives on small islands in the middle of the ocean. It is a herbivore that eats the leaves of trees. It can grow up to five meters tall, and has long, thin legs. It can use its long legs to walk across the water to other islands through shallow water. Even though it is a gentle creature, the Treestrider is sometimes feared because of its size.

Balloonbug

The Balloonbug is a small insect that can inflate itself like a hot air balloon. It is red-orange in color. The Balloonbug is not very fast, but it can stay in the air for long periods of time. It feeds on small mosquitos. When it is full, it will deflate itself and fall to the ground. It will then crawl back to its nest. Pregnant females will sometimes inflate themselves and float away to find a new home.

Tundran

The Tundran is a huge mammal that roams the coldest parts of the world. The Tundran is a massive creature, reaching up to 4 meters in height and weighing over 2 tonnes. The Tundran's fur is thick and white, insulating it from the cold weather. It has two large, curved horns, which are used to dig through the snow in search of food. Tundrans are very gentle creatures. The Tundran is a sacred creature in many cultures. In Inuit mythology, the Tundran is the spirit of a lost child, who roams the world looking for its mother. The Tundran is also considered to be a bringer of good luck in many cultures.

Luzak

The Luzak is a wolf-like creature that lives in forests and mountains. It is a good climber, and can climb almost any surface. It is the size of a wolf, and has dark gray fur with some blue in it. Like wolves, Luzaks are known to howl at the moon. Unlike wolves, however, Luzaks are solitary creatures, and will only come together to mate. They are very good hunters, and will often hunt small mammals and birds. They are not commonly seen, and are considered to be very rare.The Luzak is a member of the canidae family, which includes dogs, wolves, and foxes. It is related to the maned wolf. They are the only known species of canid to have blue fur.

M'boro

The m'boro is a large monkey that lives in the rainforest. It has orange and yellow fur, and a long tail. They are good climbers, and will often eat fruits and insects. M'boros are not dangerous to humans, but they can be aggressive if they feel threatened. M'boros are native to the rainforests and require a lot of space to move around. They have a diet that includes both fruits and insects. Oftentimes, groups of m'boros will work together to find food. The m'boro's loud calls can often be heard throughout the forest.M'boros communicate with each other using a variety of sounds, including screams, grunts, and barks. They also use their tails to communicate, by waving them back and forth.

Gleep

The Gleep (Gliphus fastus) is a small mammal in the family Gliphus, that lives in grasslands. The adult Gleep can grow to the size of a large rat. They have a thick coat of fur that is gray in color, with black stripes running along their back. Their belly is a blue color, and they have a long, fuzzy tail. Gleeps have large ears which they use to listen for predators and prey. They are very curious and like to explore their surroundings. Gleeps are also very good at running and jumping. They are often kept as pets because of their playful and friendly nature.The Gleep is a herbivore, and feeds on grasses and other plants. They are active during the day, and sleep in burrows at night. Gleeps live in small family groups, and are very social animals. They are also known to be very good swimmers.

Imag

The "Imag" is a large cat that lives in grasslands. It has a short fur that is yellowish-brown in color and patterned with black stripes and a white chest. It is a skilled hunter, able to stand perfectly still and blend in with its surroundings – making it difficult for prey to spot it. The Imag’s diet consists mainly of small animals such as rodents and birds. Females give birth to litters of up to four kittens, which they raise on their own. Interestingly, the Imag is one of the few large cats that does not roar – instead, it makes a vocalization known as a “chuff”.

Lumander

The lumander is a species of salamander found in humid forests. It is notable for its luminescent properties, which give it a pale blue-green appearance. The lumander is also notable for its antlers, which are used to attract mates. The lumander is a nocturnal creature, and is most active during the night. It is an ambush predator, and will often lie in wait for its prey. When an unsuspecting victim comes close, the lumander will lunge out and grab it. The lumander is a relatively small salamander, measuring only about 10 centimeters in length. It is believed to have a lifespan of around 10 years.

Foost

The foost (also known as the arctic komodo) is a large white lizard-like creature that inhabits the arctic regions. It is closely related to the komodo dragon, and is thought to be its white cousin. The foost is a solitary creature, and is rarely seen by humans. It is a scavenger, and feeds on carrion and seals. The foost is a massive creature, growing up to 3 metres (10 feet) in length. It has a large head with a big mouth, and sharp teeth. It has a long body, and short legs. The foost is covered in white fur, which helps it blend in with its snowy surroundings. It is the only reptile known to live in the arctic.

Screeper

A screeper is a type of spider that produces very thick, rope-like webbing. They are typically found in dark places, such as caves or under rocks. Caves and other dark places provide the perfect habitat for these spiders, as they are protected from the sun and other predators. The webbing of the screeper is unusally thick, because it is used to catch large prey. The thick webbing is also used to create cocoons in which the spider can lay its eggs. When the eggs hatch, the young spiders will climb to the top of the cave or rock, where they will begin to build their own webs.

Tromb

The "Tromb" is a creature that lives near large bodies of water. It has an extension on its head that can be rolled out, like the tongue of a frog, to catch fish. Because of this it does not need to enter the water to hunt, and can often be found perched atop rocks near the shoreline. It is a shy creature and will often hide in the shadows when humans are near. The Tromb is also known to sing a beautiful, haunting song that can be heard for miles around.

Fosha

The Fosha is a small, blue-furred creature found in forests. It is a peaceful creature, and is often seen looking up at the stars. It has long ears, and is fox-like in appearance. It eats mainly berries and insects. They live in dens made from sticks and leaves, and line them with soft grasses.Fosha mate for life, and often have large families. Despite their gentle nature, Fosha are fierce protectors of their family. They will fight fiercely if they or their loved ones are threatened.

Hundle

Hundles live in the mountains. They are hunters with a thick fur that protects them from the cold. When hundles have cubs they often have to leave them alone for long stretches of time to hunt. Since hundle cubs are helpless this is a dangerous time for them.Hundles are very good at tracking and finding prey. They are also very good at climbing using their claws. If a hundle is threatened they will usually try to run away but if they are backed into a corner they will fight back with their claws and teeth. After a hundle eats they will often curl up into a ball to sleep for a long time.

Osk

The "Osk" is a very tall standing bird that can grow up to 3 meters in height. Because it only has small wings it cannot fly. It has a golden color with a blue "collar" around its neck. They live in dry grasslands. An osk can run up to 70 kilometers per hour. Fruits and small animals are their primary food source, but they will also eat insects, lizards, and rodents. They have a large extension on their head that is used to defend themselves and impress potential mates. This extension amplifies their calls, which can be heard up to 5 kilometers away.

Waterwyrm

The waterwyrm is a giant snake-like creature that is said to live in the oceans. It is said to be as large as a whale, and to have a body that is covered in scales. The waterwyrm is said to be a fearsome predator, and to be able to kill and eat any creature that it comes across. The largest waterwyrm ever recorded was said to be over 30 meters long, and was said to have been killed by a group of fishermen in the early 1900s.

Yclyndar

The Yclyndar is a large predatory bird with purple feathers. It is found in high mountain ranges. It has a long, curved beak and sharp talons. It is an agile flier and can reach speeds of up to 60 miles per hour. The Yclyndar feeds on other birds and small mammals. It has excellent eyesight and can see small prey from up to a mile away. It nests in caves or on ledges. Its plumage is highly prized by collectors.

Alwari

The Alwari is a very social creature, and is often seen in groups. It is a herbivore, and its diet consists mainly of plants and roots. It lives in dense forests where it can hide well. The Alwari is known for its intelligence, and is able to communicate with other members of its species using a complex system of grunts and body language. The Alwari is also known to be a very good swimmer, and is often seen swimming in rivers and lakes. The Alwari's horns are used for digging up food, but also for self defence and to intimidate rival Alwari.

Hydrofant

The Hydrofant is a large fish found in large freshwater lakes. It is called Hydrofant, because it can grow as large as an elephant. It is a very slow-moving fish and is not very agile, but it can use its large mouth to vacuum up smaller fish and crustaceans. It lays eggs which hatch into larvae. The larvae grow into adults in about two years. The larvae of the Hydrofant are very small and live in the water column. They feed on plankton and other small organisms. The Hydrofant can live for up to 20 years in the wild.

Horned furtoad

The horned furtoad is a toad that lives in the rainforest. It is covered with a moss-like fur that is red-brown in color, and it has many extensions that look a bit like coral. The red extensions on its head are used by the males to attract a mate. The toad is nocturnal and spends most of its time hiding in trees or under rocks. It eats insects, spiders, and other small animals. When it is threatened, it can inflate its body to make itself look bigger and more intimidating. It can grow to about 15 cm in length.

Chimawk

The Chimawk ( blackbirdus magnificus) is a large bird found in cold forests in the northern hemisphere. It is easily distinguished by its black feathers and red head, as well as its large beak. The Chimawk is a non-flying bird, due to its small wings. It is a timid creature that usually lives in small groups of no more than 10. The Chimawk's diet consists mainly of small rodents, which it hunts, as well as berries.

Introduction

Ever since I started playing guitar some 10 years ago I dreamt of writing a song of my own someday that I can be proud of. Sometimes a nice melody or chord progression pops into my head, but somehow the lyrics that I write always feel a bit uninspired.

Then I joined a team for the AI Song Contest in 2021, and we wrote a complete song with the help of AI. We had a process of turning sci-fi stories into lyrics using GPT-3 prompting. This resulted in pretty bizarre, but very interesting lyrics. However I felt this method lacked a feeling of control. I wanted to be able to influence the style of the music, the mood and the topic the song was written about. So I continued the lyrics generation project.

In this blogpost I will explain different techniques I used to generate lyrics with the help of GPT-3.

• Generating lyrics using prompts
• Generating lyrics using a finetuned GPT-3 model
• Creating variations for existing lyrics
• Conclusion.

Generating lyrics using prompts

GPT-3 is created to finish a text given a certain input. These text inputs are also called prompts. The easiest way to use this model is to write a prompt that triggers the model into writing the text you want. For example, if you want the model to write a good joke, you could prompt it with the following text: "A very funny joke:". Prompting works better when you give some examples of what you want first, so GPT-3 can use the pattern in your examples to figure out what you want to achieve.

I first prompted the model without giving it any examples, to write a song about following your dreams. Bold text is the prompt, and normal text is the output that GPT-3 generated.

The result is pretty much what I asked for, but it is very basic. There are no metaphors, and the subject is almost copied literally into the lyrics. But the cool thing is that GPT-3 understood that a song should consist of different verses.

To generate more lyrical results I tried adding some examples to the prompt, in the hope that this would make the lyrics a bit more poetic. This allows GPT-3 to mimic the structure of the examples in the generated text. The prompt with examples is structured as follows

description: example description 1
lyrics: example lyrics 1 
<---> 
description: example description 2
lyrics: example lyrics 2
<---> 
description: example description 3
lyrics: example lyrics 3
<---> 
description: description for the new song to be generated
lyrics:

In between the examples I added a divider, to make it easier for the model to understand the structure I am looking for.

To make it easier to prepare the examples we can also use GPT-3 to generate descriptions of lyrics, by prompting it with the lyrics and asking it to describe them. An example:

The GPT-3 API has a temperature slider that controls how much randomness to allow. Temperature 0 means no randomness is allowed, and only the most probable characters are selected. A temperature of 1.0 is the highest and means a lot of randomness is allowed. This creates more interesting twists in the text, but also makes the texts more chaotic. The default value for temperature is 0.7. I found that increasing this value to 0.8 meant that the generated lyrics were a bit more interesting and poetic.

The following lyrics are created using the same prompt, but with three examples and a temperature value of 0.8 instead of 0.7. The examples are the songs

• Walk away by Franz Ferdinand
• Guaranteed by Eddie Vedder
• Universally Speaking by the Red Hot Chilli Peppers

For the next lyrics I added two more example songs and adjusted the temperature to 0.9, to make the lyrics even more imaginative.

The example songs added are:

• Sunsets for somebody else by Jack Johnson
• The hardest part by Coldplay

It seems that GPT-3 understood the concept of verses now, and it even starts to rhyme a little. See the first verse where "dream" rhymes with "mean", and "stone" rhymes with "own".

There maximum length of prompt and output in the latest GPT-3 model is 4000 characters, and this prompt with output uses up a little over 2000 characters. So I will add two more examples to see if there is more improvement.

The songs added are:

• Stop this train by John Mayer
• Dream catch me by Newton Faulkner

Interestingly if you prompt GPT-3 to answer questions and add "Let's think step by step" the accuracy of the answers increases by a lot.

Here's the leaderboard of prompts to add to GPT-3.
Can you guys come up with anything better? https://t.co/bGnBroMdR5 pic.twitter.com/a3JCdUjlYu

— Aran Komatsuzaki (@arankomatsuzaki) May 25, 2022

We can do something similar by adding tags like [Verse], [Verse variation], [Chorus] and [Bridge] to the examples. This makes the model "think" of the structure of the song, because it has to add a tag before each part. An added benefit is that the tags give you more control over the structure of the generated song. If you want to add another verse you can simply add a [Verse variation] tag, and see what the generator fills in.

An example of a song generated with added tags.

Generating lyrics using a finetuned GPT-3 model

Another way to introduce more of the desired lyrical structure and poetic quality into the generated texts is to fine-tune a model using the OpenAI API.

An important decision is the structure of the data you want to fine-tune the model on. I wanted to have control over the output of the generated songs by providing a description, name and bio of an artist, a mood and a song title. Using this method you have a lot of control over the generated lyrics. In addition to control over the subject of the song, you can decide to generate songs in the style of a given artist, and decide if it should be a happy or a sad song.

In the documentation they recommend that you should provide at least 500 examples for a good quality fine-tuned model. It is a lot of work to write song descriptions and bio's for at least 500 songs, so I created a python script to generate some of the data for me. As input I would provide just the artist name, the song title and the lyrics. The python script would call the OpenAI api to append the json with an artist bio, mood and a description of the provided lyrics. This is done using just the GPT-3 davinci model and a simple prompt.

The prompt for generating the lyrical description is as follows:

{lyrics} 

How would you describe the meaning of the above lyrics? Describe some of the metaphors. 

The prompt for generating the artist bio's:

Write a short bio for {artist}. What is the spirit of their music, their genre, and how would you describe their personality?

The prompt for describing the mood of the lyrics:

{lyrics}

What is the mood of the above lyrics?

The mood is

An example of the output generated by this script. The "bio", "description" and "mood" fields are filled using GPT-3.

Using descriptions like this for 500 songs I fine-tuned a GPT-3 Davinci model. The Davinci model is the largest and most capable of the GPT-3 models available. The larger the model, the more emergent capabilities appear. For example, the smallest model can do things like parsing text, simple classification, address correction, coming up with keywords. Capabilities like summarizing text, finding out cause and effect, and creativity only appear in the larger models. Fine-tuning a model like that for 2 epochs took just a few minutes and cost around $20.

This is a subset of my fine-tuning file, containing 4 out of the 500 songs I used for the model. It should be in json lines format, which is one json per line. Note that \n\n###\n\n is used to indicate this is where the completion starts, and <!> is used to indicate the end of the lyrics. It can be used as a stop condition when generating lyrics.

{"prompt": "artist: Black Keys\nbio: The Black Keys are an American rock band formed in Akron, Ohio, in 2001. The group consists of Dan Auerbach (guitar, vocals) and Patrick Carney (drums). The band's raw blues-rock sound draws heavily from Auerbach's guitar playing and Carney's minimalist drumming. Although they have been compared to other garage rock revivalists such as the White Stripes and the Strokes, the Black Keys have a more soulful and vintage sound.\ndescription: The lyrics are about someone offering to be a constant source of support and guidance for another person.\nmood: The mood is hopeful and reassuring\n\n###\n\n", "completion": " [Everlasting light]\n\nLet me be your everlasting light\nThe sun when there is none\nI'm a shepherd for you\nAnd I'll guide you through\nLet me be your everlasting light\n\nLet me be your everlasting light\n(Shoo-shoo-shoo-wa)\nI'll hold and never scold\n(Shoo-shoo-shoo-wa)\nIn me you can confide\nWhen no one's by your side\nLet me be your everlasting light\n(Shoo-shoo-shoo-wa)\n\nOh baby, can't you see\nIt's shinin' just for you\nLoneliness is over\nDark days are through\nThey're through\n(Shoo-shoo-shoo-shoo-shoo-shoo-shoo-wa)\n\nLet me be your everlasting light\n(Shoo-shoo-shoo-wa)\nA train goin' 'way from pain\n(Shoo-shoo-shoo-wa)\nLove is the coal\nThat makes this train roll\nLet me be your everlasting light\nYeah\n\nLet me be your everlasting light\nLet me be your everlasting light\nLet me be your everlasting light\nOhh...<!>"}
{"prompt": "artist: Arctic Monkeys\nbio: Arctic Monkeys is an English rock band formed in Sheffield in 2002. The group consists of Alex Turner, Matt Helders, and Nick O'Malley. The band's debut album, Whatever People Say I Am, That's What I'm Not, was released in 2006 and became the fastest-selling debut album by a band in UK chart history. The band's second album, Favourite Worst Nightmare, was released in 2007. The band's third album, Humbug, was released in 2009. The band's fourth album, Suck It and See, was released in 2011. The band's fifth album, AM, was released in 2013. The band's sixth album, Tranquility Base Hotel & Casino, was released in 2018.\ndescription: The lyrics of the song seem to be criticizing the pretentiousness of some people in the indie music scene. The singer is talking about how some people try to act like they are from San Francisco or New York City, when they are actually from much smaller towns. They also seem to be making fun of how some people in the scene try to act like they know everything about music, when they really don't.\nmood: The mood is one of disdain and mockery\n\n###\n\n", "completion": " [Fake tales of san francisco]\n\nFake Tales of San Francisco\nEcho through the room\nMore point to a wedding disco\nWithout a bride or groom\n\nThere's a super cool band yeah\nWith their trilbies and their glasses of white wine\nAnd all the weekend rock stars in the toilets\nPracticing their lines\n\nI don't want to hear you\n(Kick me out, kick me out)\nI don't want to hear you no\n(Kick me out, kick me out)\nI don't want to hear you\n(Kick me out, kick me out)\nI don't want to hear you\nI don't want to hear your\n\nFake Tales of San Francisco\nEcho through the air\nAnd there's a few bored faces at the back\nAll wishing they weren't there\n\nAnd as the microphone squeaks\nA young girl's telephone beeps\nYeah she's dashing for the exit\nOh, she's running to the streets outside\n\"Oh you've saved me,\" she screams down the line\n\"The band were fucking wank\nAnd I'm not having a nice time\"\n\nI don't want to hear you\n(Kick me out, kick me out)\nI don't want to hear you no\n(Kick me out, kick me out)\nYeah but his bird said it's amazing, though\nSo all that's left\nIs the proof that love's not only blind but deaf\n\nHe talks of San Francisco, he's from Hunter's Bar\nI don't quite know the distance\nBut I'm sure that's far\nYeah, I'm sure it's pretty far\n\nYeah, I'd love to tell you all my problem\nYou're not from New York City, you're from Rotherham\nSo get off the bandwagon, and put down the handbook\nYeah, yeah, yeah, yeah, yeah\n\nGet off the bandwagon and put down the handbook\nGet off the bandwagon and put down the handbook\nGet off the bandwagon and put down the handbook\nGet off the bandwagon and put down the handbook, yeah<!>"}
{"prompt": "artist: Coldplay\nbio: Coldplay is a British rock band formed in 1996 by lead vocalist and pianist Chris Martin and lead guitarist Jonny Buckland at University College London. After they formed under the name Pectoralz, Guy Berryman joined the group as a bassist and they changed their name to Starfish. Will Champion joined as a drummer and backing vocalist, completing the lineup. Coldplay achieved worldwide fame with the release of their single \"Yellow,\" followed by their debut album, Parachutes (2000), which was nominated for the Mercury Prize.\ndescription: The lyrics seem to be about wanting a simple, down-to-earth love, without any pretence or artifice. The speaker is tired of reading about mythical heroes with superhuman gifts, and instead wants something (and someone) more real and relatable.\nmood: The mood is romantic and longing\n\n###\n\n", "completion": " [Something like this]\n\nI've been reading books of old\nThe legends and the myths\nAchilles and his gold\nHercules and his gifts\nSpiderman's control\nAnd Batman with his fists\nAnd clearly I don't see myself upon that list\n\nBut she said, \"Where'd you wanna go?\nHow much you wanna risk?\nI'm not looking for somebody\nWith some superhuman gifts\nSome superhero\nSome fairytale bliss\nJust something I can turn to\nSomebody I can kiss\nI want something just like this\"\n\nDoo-doo-doo, doo-doo-doo\nDoo-doo-doo, doo-doo\nDoo-doo-doo, doo-doo-doo\nOh, I want something just like this\nDoo-doo-doo, doo-doo-doo\nDoo-doo-doo, doo-doo\nDoo-doo-doo, doo-doo-doo\n\nOh, I want something just like this\nI want something just like this\n\nI've been reading books of old\nThe legends and the myths\nThe testaments they told\nThe moon and its eclipse\nAnd Superman unrolls\nA suit before he lifts\nBut I'm not the kind of person that it fits\n\nShe said, \"Where'd you wanna go?\nHow much you wanna risk?\nI'm not looking for somebody\nWith some superhuman gifts\nSome superhero\nSome fairytale bliss\nJust something I can turn to\nSomebody I can miss\nI want something just like this\nI want something just like this\"\n\nOh, I want something just like this\nDoo-doo-doo, doo-doo-doo\nDoo-doo-doo, doo-doo\nDoo-doo-doo, doo-doo-doo\nOh, I want something just like this\nDoo-doo-doo, doo-doo-doo\nDoo-doo-doo, doo-doo\nDoo-doo-doo, doo-doo-doo\n\nWhere'd you wanna go?\nHow much you wanna risk?\nI'm not looking for somebody\nWith some superhuman gifts\nSome superhero\nSome fairytale bliss\nJust something I can turn to\nSomebody I can kiss\nI want something just like this\n\nOh, I want something just like this\nOh, I want something just like this\nOh, I want something just like this<!>"}
{"prompt": "artist: Counting crows\nbio: Counting Crows is an American rock band from Berkeley, California, formed in 1991. The band consists of Adam Duritz (lead vocals, piano), David Bryson (guitar), Charlie Gillingham (keyboards, piano, accordion), Dan Vickrey (guitar), David Immerglück (guitar, banjo, mandolin), Matt Malley (bass) and Jim Bogios (drums). Counting Crows gained popularity following the release of their debut album, August and Everything After, in 1993. Their hit single, \"Mr. Jones\", peaked at number five on the Billboard Hot 100. The band's influences include Van Morrison, R.E.M., Bob Dylan, and The Band.\n\nThe spirit of Counting Crows' music is one of melancholy and introspection, with Duritz's lyrics often focusing on relationships and heartbreak. The band's musical style has been described as a mix of alternative rock, pop, folk, and country. Counting Crows' personality has been described as laid-back and easy-going. The band is known for their live performances, which often feature extended jams and impromptu cover songs.\ndescription: The lyrics appear to be about a man who is planning to take a trip to Spain with a girl. He is leaving his old life behind him, and is looking forward to starting fresh in a new place. The lyrics could also be interpreted as being about a relationship that is coming to an end. The man is ready to move on, and is hoping to find someone new to start a new chapter with.\nmood: The mood is carefree and happy\n\n###\n\n", "completion": " [Holiday in spain]\n\nGot no place to go\nBut there's a girl waiting for me down in Mexico\nShe's got a bottle of tequila\nA bottle of gin\nAnd if I bring a little music I can fit right in\nWe've got airplane rides\nWe've got California drowning out the window side\nWe've got big black cars\nAnd we've got stories how we slept with all the movie stars\nI may take a holiday in Spain\nLeave my wings behind me\nDrink my worries down the drain\nAnd fly away to somewhere new\nHop on my choo-choo\nI'll be your engine driver in a bunny suit\nIf you dress me up in pink and white\nWe may be just a little fuzzy 'bout it later tonight\nShe's my angel\nShe's a little better than the one that used to be with me\n'Cause she liked to scream at me\nMan, it's a miracle that she's not living up in a tree\nI may take a holiday in Spain\nLeave my wings behind me\nDrive this little girl insane\nAnd fly away to someone new\nEverybody's gone\nThey left the television screaming that the radio's on\nSomeone stole my shoes\nBut there's a couple of bananas and a bottle of booze\n\nOh well, Happy New Year's, baby\nWe could probably fix it if we clean it up all day\nOr we could simply pack our bags\nAnd catch a plane to Barcelona 'cause this city's a drag\nI may take a holiday in Spain\nLeave my wings behind me\nFlush my worries down the drain\nAnd fly away to somewhere new\nTake a holiday in Spain\nLeave my wings behind me\nDrive this little girl insane\nFly away to someone new\nFly away to someone new\nFly away to someone new<!>"}

Let's see if the fine-tuned model is better at generating lyrics. Let's try generating lyrics in the style of a few different artists.

Because the model kept generating existing song title's for the artists, I created an extra prompt to generate these. The title generating prompt is very simple:

Come up with an original title for a song about following your dreams by {artist}.

Below are some songs generated by prompting the fine-tuned model with different artists. I cherry picked the songs by generating up to 5 songs per prompt, and choosing my favourite.

"Chasing After Dreams", in the style of Jack Johnson.

I liked these lyrics a lot, so I added some chords and recorded a simple version of the song.

"The ghost of you", in the style of the Red Hot Chili Peppers.

"Find your way", in the style of John Mayer

"Dreams of the remaining", in the style of Queens of the stoneage

"Clouds on the horizon", in the style of Franz Ferdinand

"Chasing the sun", in the style of Muse.

"A whirlwind of dreams", in the style of Coldplay

To see if the model can also generate songs in a completely different style. Here is "Don't you give up on me", in the style of Taylor Swift.

Generating variations on existing lyrics

Sometimes you have generated a good verse, but GPT-3 decides that repeating the same verse over and over is a good idea. We can use prompts to solve this issue and introduce variation in the lyrics.

To do this we have to give some examples of a verse, and an alternative wording in the same structure. For example:

lyrics: 

On the corner of Main Street
Just trying to keep it in line
You say you wanna move on and
You say I'm falling behind
Can you read my mind?
Can you read my mind?

variation on the above lyrics:

I never really gave up on
Breaking out of this two-star town
I got the green light, I got a little fight
I'm gonna turn this thing around
Can you read my mind?
Can you read my mind?

For this experiment I added seven such examples to the prompt in the format

lyrics:

{example lyrics 1}

variation on the above lyrics:

{example variation 1} 

---

lyrics:

{example lyrics 2 - 7}

variation on the above lyrics:

{example variation 2 - 7}

--- 

lyrics:

{lyrics you want to generate a variation on}

variation on the above lyrics:

If I take the first verse from the generated Franz Ferdinand song and put it in there it outputs the following variation.

Inserting this new variation gives a whole new spin to the song

Without retraining the fine-tuned lyrics model we can steer a song in new directions by adding tags to the already created output. For the next example I thought the lyrics generated for the previously generated song "Chasing after dreams" could use a twist to keep the song interesting until the end. A bridge would be a good way to achieve this twist. To get the model to write a bridge I added tags for verse, chorus and pre-chorus and ended the prompt with a bridge tag. Amazingly it did add a bridge, even though the model was not trained to work with these tags! I think this is a great example of the flexibility of these models.

Conclusion

Although it still takes some of tweaking to get the results you want, this fine-tuned version of GPT-3 is very useful for providing inspiration for lyrics. It often comes up with ideas that I would never have thought of on my own, and it's a great way to get the creative juices flowing. The process of generating new lyrics and trying different styles is a lot of fun, and I found myself trying many different variations. Prompt programming is very flexible so it is easy to try out new ideas.

However one should be careful not to plagiarize other lyrics or writings. for example the prompt with 5 examples used some lyrics that are taken from an inspirational quote from Liam Payne.

Dreams are like stars
You may never touch them
But if you follow them
They will lead you to your destiny

I was also surprised at how useful GPT-3 is out of the box for preparing data. You can ask it to extract useful features from text, and use these features as data to finetune a new model.

I can definitely see tools like this being used by writers or musicians. Maybe not to generate entire songs with one click of a button, but more as a writing companion.

Introduction

At the start of April a friend called me to ask me if I wanted to participate in a competition called the "AI Song Contest". The goal of the contest is to bring musicians and data scientists together to create music using Artificial Intelligence. Last year, in 2020, the first edition of this contest was organised. We would have about a month to create everything needed to generate a song, so the musicians would have enough time to record and produce. I had some experience in machine learning, but none in generated music, so the deadline was intimidating. But after thinking about it for a minute I decided to go for it. I wanted to learn more about transformer models that seemed to have all these magical capabilities, and to learn about them while making music at the same time seemed perfect. I like to play guitar and always felt the urge to write my own songs. Also at that time I had just been granted access to the OpenAI Api, so I could put GTP-3 to the test with this challenge. I decided to take a week off from work, rent a chalet and focus completely on this challenge.

You can find our page on the AI Song Contest website:

https://www.aisongcontest.com/participants/lovelacethemachines-2021

Our team consisting of colleagues, friends and family who all hail from The Netherlands. The team started out with two colleagues and the whole thing just snowballed into a full-fledged team. Our data scientists are: Hennie, played Xylophone and piano for over 10 years and likes to learn from these type of contests; Rick, humanities major and notoriously a-musical; Salomon, jazz and piano player in a big band; Vincent, guitar player and always wondering how things work. And our musicians: Joëlle, singer-songwriter that ditched the whole music part to become just a writer; Thomas, composer and specialized in writing for film and games; Wendel, wellness manager by day, DJ by night (which still has something to do with wellness really). We set out on a dual quest: to find out if AI can become a new band member and to test some nice algorithms we don’t get to use during working hours.

The challenge

The next week we had our first team meeting. We called ourselves "Lovelace & The machines", in reference to Ada Lovelace who was already thinking about machines that could compose music in the 19th century. She was far ahead of her time in thinking about computers, or Analytical Engines as she called them. In her notes she wrote:

[The Analytical Engine] might act upon other things besides number, were objects found whose mutual fundamental relations could be expressed by those of the abstract science of operations, and which should be also susceptible of adaptations to the action of the operating notation and mechanism of the engine...Supposing, for instance, that the fundamental relations of pitched sounds in the science of harmony and of musical composition were susceptible of such expression and adaptations, the engine might compose elaborate and scientific pieces of music of any degree of complexity or extent.

We started discussing the challenge ahead of us and the problems we needed to tackle to get there. The AI Song Contest did not have many requirements, except for the requirement to be creative and put AI to good use in the process of creating our song. We decided to do as much with AI as possible, and only give human input where it is necessary.

The following challenges needed to be tackled:

• Writing the chord sequences and melody
• Writing the lyrics
• Creating drum and bass lines
• Putting it all together into a song

Before we could decide on a strategy we first had to figure out what the existing state of the art methods were.

Existing research

There has been a lot of progress in music generation in the past few years. I will mention some of the inspiration we had when creating our song.

Magenta: Music Transformer

Since I was interested in learning more about transformers I set out to find out if transformer networks could be used to generate music. Since these networks excel in finding underlying structure of language, it seems like they might also be a good fit for figuring out the structure of music.

And indeed, I stumbled upon research by Google Magenta, the music transformer. The music transformer uses midi like events as input and manages to create beautiful and expressive piano pieces.

Transformer networks use a mechanism called attention to find patterns in data. When generating a new note, instead of looking at the whole piece of music as equal it learns to identify the parts that influence what this note should be. How this attention mechanism functions when generating a song is beautifully visualised in Magenta's blogpost.

Magenta: MusicVAE

Another technique that might be interesting for generating music is the auto-encoder. An auto-encoder gets raw data as input, pushes it through a neural network with a small layer in the middle, and tries to reconstruct the raw data in the output layer again. By pushing the data through this small layer in the middle you force the network to figure out how to compress the data. The space of possible compressed representations is called the latent space.

In the Magenta MusicVAE a variational auto-encoder is used. Instead of representing the data as a vector it is represented as a mixture of distributions. This adds the property that you can make smooth interpolations between two points in the latent space. This is very cool, because if you would interpolate between two musical samples all the samples in between would also sound good.

OpenAI Jukebox

We found that OpenAI had been doing some super exciting work in this direction. They created a neural network called Jukebox that generates music as raw audio, and can even add vocals to the generated music based on lyrics that are added as input data. They use a VAE (Variational autoencoder) to compress raw audio and lyrics together to a compact format that captures the characteristics of this combination. Then a transformer network is trained to generate new audio in this compressed space. The generated data is then converted back into raw audio. This direction is super promising and exciting, but we lack the compute to reproduce these results ourselves. The Jukebox model was trained on 256 V100 GPUs for 3 days, whereas our compute budget was 1 GPU.

You can explore all the generated samples here: https://jukebox.openai.com/

OpenAI MuseNet

Another direction OpenAI has ventured into was generating symbolic music using MuseNet. MuseNet works in a similar fashion as novel language models. It uses a sparse transformer network to generate pieces with a maximum of four minutes and supports 10 different instruments. Midi files encoded as a sequence of words are used as input. The input looks like this:

bach piano_strings start tempo90 piano:v72:G1 piano:v72:G2 piano:v72:B4 piano:v72:D4 violin:v80:G4 piano:v72:G4 piano:v72:B5 piano:v72:D5 wait:12 piano:v0:B5 wait:5 piano:v72:D5 wait:12 piano:v0:D5 wait:4 piano:v0:G1 piano:v0:G2 piano:v0:B4 piano:v0:D4 violin:v0:G4 piano:v0:G4 wait:1 piano:v72:G5 wait:12 piano:v0:G5 wait:5 piano:v72:D5 wait:12 piano:v0:D5 wait:5 piano:v72:B5 wait:12

The model is trained to predict the next character in the sequence, like a language model, and by doing so repeatedly it generates music.

A really cool thing is that you can try out MuseNet on the OpenAI website to generate your own samples!

Pop music transformer

When looking for open implementations of the music transformer we found the Pop Music Transformer. As in the music transformer model they use a midi-like representation for their input data too, but they created a new format; REMI. REMI stands for REvamped MIDI-derived events and adds more structure to the data by dividing music into measures and beats, and it allows for chords to be added to the training data.

Instead of using a standard transformer they use the transformer XL. A normal transformer is restricted in the maximum number of characters it can evaluate at once, because the attention mechanism used is very expensive. It scales exponentially, so looking at 1024 characters instead of 512 is not a doubling of necessary compute but takes four times as much, looking at 2048 characters takes 16 times as much and so on. This means that a midi that is used as input is cut into parts, and the model can only learn the relations inside a single part. This means that the model can't learn how a complete song is structured. Transformer XL uses a trick in which it remembers a compressed representation of previous parts so it does not lose its context when generating music. This makes it possible to generate longer musical pieces that are still coherent.  

Aiva

One of the music generation tools we looked at is Aiva. Aiva is a very polished tool for creating music based on your own input. You can upload a midi file and create new music based on this input in different styles and using different instruments. They don't fully disclose how they achieve their results, but they sound very impressive. In this blogpost they say they use a combination of genetic algorithms and deep neural networks.

Although Aiva is a really impressive tool, we decided not to use it because it takes away the effort and understanding of the algorithms on our part.

OpenAI language models: GPT-2 and GPT-3

For generating lyrics we looked into the GPT-2 and GPT-3 models, since these achieve state of the art results in language generation.

The GPT models are transformers trained to predict the next character given a sequence of characters. GPT-2 had 1.5 billion parameters in it's neural network, and was trained on a dataset consisting of the text from 8 million web pages. For GPT-3, the next iteration of the model, they scaled this up to 175 billion parameters. The more parameters the better the model would be able to capture patterns in language, and thus the better it would be at predicting the next character in a text. But when scaling the model up the researchers at OpenAI discovered something interesting. Not only did the models learn to predict characters, they also learned how to answer questions or perform actions on text given a few examples. (see https://arxiv.org/abs/2005.14165)

Process overview

After diving into the existing methods we are ready to get our hands dirty.

Before explaining the process in depth I first want to share an overview image. Since our team worked on so many different things, talking about all of them would not do them justice. Instead I will dive deeper into the highlighted parts, that I was involved in. These include generating music, lyrics and variations on generated music.

Chords & melody

For generating melodies we chose to work with the pop music transformer. The code for this project was available on Github, and the authors wrote a paper describing their method.

The pop music transformer builds on the idea of generating music with transformers, but they add more structure to the input data to be able to generate songs with more rhythmic structure. Furthermore they use Transformer XL to generate music that sounds consistent for longer periods of time.

REMI Format

To accomplish this the authors introduce the REMI format, which stands for REvamped MIDI-derived events. This format is a way to represent music in the MIDI format as text. A MIDI file represents a piece of music using a number of tracks. The tracks can each be assigned an instrument and the notes that are played by that instrument. There are 128 instruments and every instrument can play 128 different pitches. For every note that is played it denotes which pitch is played, the precise start and end time of the note and how loud it is. Since there is not really a concept of beats and bars in this format, a model learning how to make music based on this raw data would have to discover that music is usually structured in this rhythmic pattern. The REMI format is a way of helping the model by adding this structure to the data. Instead of using exact start and end times the music is divided in bars, where every bar is divided in 16 parts to describe the starting points of the notes. For every note a duration is added. This duration has some more precision and can be described in 32th parts of a bar.

For example if you want to convert this beat from "Comptine D'un Autre Été" by Yann Tiersen to REMI it would go as follows:

It will be converted to something that looks like this:

This is then converted to text:

Bar
Position 1/16
Play midi note 64
Hold for 4x 1/32 measure
Velocity 64
Play midi note 52
Hold for 4x 1/32 measure
Velocity 64
Position 3/16
Play midi note 67 
Hold for 2x 1/32 measure
Velocity 64
Play midi note 59
Hold for 4x 1/32 measure
Velocity 64
Position 4/16
Play midi note 66
Hold for 2x 1/32 measure
Velocity 64
...

Another addition to the standard MIDI representation is the chord extraction that the REMI model uses. It looks at the notes played and tries to find the chords that go with the notes. They then add the chords as words to the input text, making it easier for the model to learn chord sequences.

Transformer model

The text is fed into an autoregressive language model that tries to find patterns it can reproduce. An autoregressive model is a model that tries to predict the future by looking at past values. Say I input a sentence that starts with "I really like listening to " and then ask an autoregressive language model to complete it it looks at the probabilities for the next character or word based on the data it is trained on. If the model was trained on my utterances it would probably continue to output "the Red Hot Chili Peppers".

A few years ago recurrent neural networks would often be used to find relationships between words. In recurrent neural networks a sentence is built word by word, and after every word a compact representation (vector) of the sentence so far is created and passed on to the next step. The next step then uses the compact representation of the current sentence + the new word to create a new compact representation that is can pass on to the next step. The advantage of this approach is that sentences of any length can be processed. The disadvantage is that it makes it really hard to preserve relationships between words that are far apart, because every word in between has to be added to this compact representation. The compact representation can only store a limited amount of information, so if two words are too far apart the context is lost.

In 2017 the paper "Attention is all you need" a new type of neural network called the Transformer was introduced. The Transformer improved this by a mechanism called attention. Instead of passing the compact representation of the sentence along every time it is now possible to look directly at other words in the sentence. There are usually several attention layers that each find bigger patterns. If you take this example from the Google AI blog you can see that the word "it" can relate to "animal" or "street" depending on the context.

The first attention layer could model the words "it" could relate to. This would probably be a noun like "animal" or "street", so it should pay extra attention to those words. The next layer could use this combined representation of "it could be related to a street or an animal" to find out which of these words is meant by "it". It should pay extra attention to "wide" or "tired" to find out more.

If you want to better understand how transformer networks work I can recommend this video.

The cool thing is that we can treat songs converted into REMI as text and use these transformer language models to find patterns.

Transformer XL

The transformer model is able to find these relationships between words amazingly well, but it still has its drawbacks. The attention step is expensive, since for every word we have to find out how much this word should pay attention to each other word. So if there are 512 words, we need to calculate attention for 512 * 512 connections between words. The number of calculations needed to be done is x^2 where x is the number of words we want to look at. If we can train a network with an attention window of 512 on one video card, we would need 4 GPUs to do 1024 words, and 16 GPUs to do 2048 words.

Of course you can split the sentences in parts that you can process, but this could deprive the model of essential context. Say you wanted to predict the next word in this sentence

"Alice and Bob went on holiday to a tropical island. Of course they prepared well and packed their ..."

But because of computational restrictions the model can only look at this part of the sentence when making a prediction:

• Of course they prepared well and packed their ...

The model would not be able to determine if it has to fill in "Snow boots" or "Swimwear and Hawaii shirts".

Transformer XL brings back the idea of the recurrent neural network to solve this problem. It makes a compact representation of the piece of text that came before and passes this to the next step when calculating probabilities. The input would then be something like:

• Sentence: Of course they prepared well and packed their ...
• Compact representation of text before: Holiday / Tropical (represented as a vector)

Then it would be way easier to say how the sentence would continue.

The same would apply to music. The compact representation of what came before could hold some information on the genre of the music, or the instruments used, the tempo and so on.

Finding data and training the model

Now we had a way of representing music and creating a model that could generate new music based on midi data. But these types of models are very data hungry, so where could we find enough data?

We quickly found the Lakh midi dataset. A huge collection of 176,581 midi files of all kinds of songs. This would definitely be enough data to feed the model.

But of course the musicians in our team wanted to make music that reflected their personal taste, and training a model on a load of random songs would probably result in something similar to whatever style is most popular. We set out to search for MIDI files for songs from "Crystal castles", "Disclosure", "Daft Punk", "The Weeknd" and "Lana del Rey" on sites like https://freemidi.org/ so we could use these to finetune the model.

Multiple instruments

The pop music transformer was designed to generate music based on piano compositions. But for creating our song we wanted to include multiple instruments and drums. And of course all the MIDI files from the Lakh midi dataset contained multiple tracks. We had to adjust the model to allow for multiple instruments.

To do this we extended the REMI representation. In the original REMI representation a note would be represented as:

Position_1/16      # Position in the bar
Note Velocity_24   # Loudness of the note
Note On_64         # Pitch of the note
Note Duration_8    # Duration of the note in 1/32 measures

In our representation this would become

Position_1/16      # Position in the bar
Instrument_33      # Instrument playing the note
Note Velocity_24   # Loudness of the note
Note On_64         # Pitch of the note
Note Duration_8    # Duration of the note in 1/32 measures

Training the model

We then proceeded to train the first model on Google Colab. The cool thing about Colab is that they provide GPUs to researchers, so we could train our models a lot faster this way. We collected a set of around 30 or 40 MIDI files that the musicians liked and fed them to our model as training data. To our surprise some samples rolled out that sounded better than we expected, and there was a lot of variety. We felt like we were on the right track!

There was one problem though. Because this first dataset was so small it happened very often that the model would imitate exact samples from the music we used as input. Since we didn't want to plagiarise any music we decide to train a model on the Lakh midi dataset. Our reasoning was that a larger dataset would make it impossible for the model to "remember" the exact songs, and generalise better. The Lakh midi dataset consists of 176,581 unique midi files, but we used the clean version of the dataset where some filtering is applied. This dataset contains only songs with a timing of 4/4, and removes some of the duplicates or unknown songs in the set. This still leaves us with. 21,425 midis for our model to train on.

For training this bigger model we set up a virtual machine on Google Cloud with a GPU and Jupyter notebook installed. When registering with Google Cloud you get $300 of budget to spend on trying out the platform. We used the entire budget to train the model. When the model was trained we set it to work generating samples. There was a lot of variety in the samples! You can listen to some of them here.

It seemed that the model learned how to combine the different instruments in interesting ways, but now the samples would not stay coherent for long periods of time. There would be nice parts of 10 to 30 seconds, but then everything would descend into complete chaos again.

Preprocessing the data

Maybe all 128 MIDI instruments was a bit much for the model to learn. And most of the instruments are not completely dissimilar. For example it would not make a big difference in notes played if you had an "Acoustic Grand Piano" or an "Electric Grand Piano". Also vocals in midi are played by flutes, saxophones or trombones and a variety of other instruments. There are so many instruments that have the same characteristics and play the same musical patterns, but the model has to learn the patterns for each of these instruments from scratch.

We decided that we would make the job of the model simpler by doing a preprocessing step on the data. We would split the music into melody, bass and accompaniment. Salomon was working on an algorithm to sort the instruments in a MIDI track based on frequency of the notes, number of notes per minute, how many notes would sound at the same time, and some other metrics. This would make it possible to automatically preprocess the Lakh dataset.

At that time I found the pop909 dataset. This dataset consists of piano arrangements for 909 Chinese pop songs. This was perfect for what we wanted to achieve, because all the songs were split into three tracks. One for the main melody (usually vocals), one for the sub-melody and one for the accompaniment (chords).

Another thing that I added at the time was transposition of the songs. It is kind of a waste of data that all the songs are in different keys. If all the songs are transposed to the same key it is easier to find connections between songs. I implemented two ways of transposing the songs. One method was to find the key of a song using Music21, a music analysis library for python, and then transposing the song until it was in the key of C (or A minor which uses the same notes). The other method was just transposing the songs to all the keys, so you would have 12 times as much training data, since there are 12 different keys.

This resulted in songs that sounded consistent for longer periods of time. Also the vocal lines were more easily distinguishable, which was convenient for the musicians who had to craft a song from the samples.

Here are some of the interesting samples generated by the model.

Finetuning to particular music

We wanted to avoid overfitting in our model, which requires large amounts of data, and at the same time steer the model in a direction that we like. To do this I downloaded MIDI files for some of the songs that we liked and extracted the two melody tracks and the accompaniment track like in the pop909 dataset. In this way we could train a model on the large dataset and finetune it on this small dataset.

You can hear that the samples are slightly more "minor" after finetuning on our songs.

Result

So now we had a large collection of generated music. The more chaotic music from the Lakh model, which was also more interesting in variety. And the music generated from the preprocessed MIDIs, which had better structure but had less surprises in it.

Lyrics

In the same time we were working on the generation of lyrics. We wanted the lyrics to sound like they were produced by an Artificial Intelligence, so they had to be futuristic. Hennie had the idea to scrape a lot of sci-fi stories from Reddit and train a GPT-2 model to generate new sci-fi texts.

The results of this model sounded like really interesting stories, but since they were not very lyrical it was hard to put them to music. Since I had just gotten access to the GPT-3 model through the OpenAI API I decided it was time to put it to work.

The GPT-3 model is a huge language model, trained on 45 terabytes of text. It is a transformer network with 175 billion trainable weights trained to predict how to continue a text given a certain text as a prompt. This huge size gives it interesting properties. It can write texts that are sometimes indistinguishable from human writing. An even more amazing property is that is can "learn" how to do new things by giving it a few examples as a prompt. I put "learn" between parentheses because giving these examples does not change the model in any way. It sees a pattern and recognises that it should continue using the same pattern to keep the text consistent.

I wondered if we could build a two-stage rocket. The first stage is the GPT-2 model generating sci-fi stories. The second stage is the GPT-3 model converting the generated stories into lyrics.

GPT-3 prompt example

I started with some simple examples. First I was wondering if the model could understand the concept of a riddle, and generate new riddles in this way.

I gave the following prompt to GPT-3 and asked it to continue writing.

Riddle: Where does a rabbit go for a shampoo?
Answer: To a hare dresser.

Riddle: Why do skeletons go on vacations alone?
Answer: Because they have no-body to go with.

Riddle: The more you take, the more you leave behind. What am I?
Answer: Footsteps.

When you would give this text to a person and asked them to continue, they would try to come up with a new riddle and put the answer below. It turned out that GPT-3 could also do this. It responded with:

Riddle: Why did the woman throw the clock out the window?
Answer: Because she wanted to see time fly.

I had some fun generating a few more of them!

Riddle: What goes up and never comes down?
Answer: Your age.

Riddle: What did the fish say when he hit a concrete wall?
Answer: Dam!

Riddle: Why did the turtle cross the road?
Answer: To get to the Shell station.

Riddle: What has teeth but can't bite?
Answer: A comb.

Riddle: How do you make a tissue dance?
Answer: Put a little boogie in it.

It also had some self-knowledge.

Riddle: What do you get when you let an AI generate riddles? 
Answer: Artificial stupidity.

Converting stories into lyrics

Then I applied the same idea to convert the generated stories into lyrics. I took some song lyrics and tried to come up with how the lyrics would look like if they were written as text.

Examples were given like this:

text:

When you are with her the night is exciting.
You go out until the sun comes up, having the time of your life. But then the daytime is a complete disaster.
In the beginning you really do your best to impress her, but then you figure out that she is not someone who sticks around.
She will go on to have new adventures, and you will just be a small part of her life.

lyrics:

She can take the dark out of the nighttime
And paint the daytime black
You will start out standing
Proud to steal her anything she sees
You will start out standing
Proud to steal her anything she sees
But you will wind up peeking through her keyhole
Down upon your knees

text:

A single lightbulb hanging overhead was casting its light on the crowd of  around ten thousand people. Everyone was communicating with each other, but they were doing so without speaking. They were communicating with their devices, and nobody dared to make any real sound.

lyrics:

And in the naked light, I saw
Ten thousand people, maybe more
People talking without speaking
People hearing without listening
People writing songs that voices never share
And no one dared
Disturb the sound of silence

Then I picked a semi-random text sample from the internet about the Mars rover, since I like reading about Mars. The input to the model was this:

As the Perseverance rover began to make tracks on the surface of Mars, a sensitive microphone it carries scored a first: the bangs, pings, and rattles of the robot’s six wheels as they rolled over Martian terrain.“A lot of people, when they see the images, don’t appreciate that the wheels are metal,” said Vandi Verma, a senior engineer and rover driver at NASA’s Jet Propulsion Laboratory in Southern California. “When you’re driving with these wheels on rocks, it’s actually very noisy.”

An interesting piece of text, but really not that interesting for putting in a song. GPT-3 then turned it into these lyrics:

I am Voyager, and I travel far.
And this is the first full song that I ever heard.
And it is the last thing that will ever be.
Baby take my hand.
It's gonna be a long long time.
Baby take my hand.
In this planet we call Earth.

I could imagine this text in a song. A song about feeling lonely and missing the place that you come from, or leaving someone and knowing you are never coming back. The most amazing thing about it to me was that it kind of made sense. The Voyager was a spacecraft sent into space to never come back, and the "song" that the text mentions could be the golden record that is on board the Voyager mission with memories of earth and music on it.

These are some of the lyrics generated by the prompt based on the sci-fi stories from the GPT-2 model.

I float in the deepest space.
The stars are black and white, the violets, blues and greens.
I'm floating where the air is raw.
My world lies far behind.
I see it in my mind.

I hope that someday it will be real.
Floating in a sea of sound,
I wonder what the people are going to think about me
on monday morning when they hear about my weekends adventures
with all these intergalactic friends of mine?

And when we sleep we dream the future.
And when we wake we have the past.
And when we're lost in time and space.
We turn our gaze to the skies.
And when we look back down to Earth.
We're all too blind to see the light.
The light is in your eyes.
The light is in your eyes.

We were caught in a quantum trap.
A digital world made of atoms.
A simulated reality made of mathematical functions.
We write the code and we use it to catch us.
We are the ones who use the particle accelerator.
We write the algorithm and we make it function.
We are the ones who simulate reality and we can also change it.

As the vessel drifts away
I can see the path we have made
Peeking through the distant haze
The long procession of our wake

I can see the places we have been
Reminding me how we used to be
Images flash before my eyes
I'm remembering when I used to be

There was only one thing I knew how to do
It was enough to get by on
Then something happened and I became aware
I became aware of the space I was in
And I knew that I was alive

And the first thing I wanted to do was
Find some way to share this with you
And as my eyes adjust to the light
I can see where we are now
I can see that we're not alone

There are signatures all around us
Reminding me how we used to be
Images flash before my eyes
I'm remembering when I used to be alive.

Creating variations on lyrics

In the same way as creating the lyrics we could use a prompt to create new lyrics based on existing ones.

As prompt I used one part of an existing song, and as continuation the next part of that song.

One of the examples I used from the lyrics of "These streets" by Paolo Nutini.

lyrics:
Where'd the days go? When all we did was play
And the stress that we were under wasn't stress at all
Just a run and a jump into a harmless fall
From walking by a high-rise to a landmark square
You see millions of people with millions of cares
And I struggle to the train to make my way home
I look at the people as they sit there alone

variation:
Cross the border, into the big bad world
Where it takes you 'bout an hour just to cross the road
Just to stumble across another poor old soul
From the dreary old lanes to the high-street madness
I fight with my brain to believe my eyes
And it's harder than you think to believe this sadness
That creeps up my spine and haunts me through the night

It turned out to be a lot harder to generate interesting variations this way. Often it would just return the same text as the input, but sometimes it would return something interesting. For example inputting these lyrics:

lyrics:
I float in the deepest space
The stars are black and white, the violets, blues and greens
I'm floating where the air is raw
My world lies far behind
I see it in my mind

Would give the next variation:

variation:
I ride the darkest planets
The moon is black and white
My world lies far behind
My feeling lies beyond
I see it in my mind

Producing the song

After we created the algorithms and generated a lot of content the musicians started working on the song. They went though all the generated music and lyrics to select their favourite parts with help of a hit prediction model created by Rick.

Then a kind of jamming process started. The musicians would create something and ask if we could create basslines, drums or variations for the selected parts. To create these variations we used MuseNet, the Multitrack Music Machine and an autoencoder model created by Salomon that could generate new melodies.

The starting point of the song was this piece that we used for the chorus.

Thomas and Wendel created the music around this starting point and Joëlle picked her favourite lyrics to sing to the melody. Joëlle started by picking some lyrics that really sounded otherworldly. Something only an AI could have come up with about us living in a simulated reality, or a Quantum Trap.

A lot of cherry picking was still required to fit all the music together. And one thing the models still lacked was the overall structure of the song. It could generate parts that sounded interesting, but it would not have repeating patterns, or a distinction between verse, bridge and chorus. So the musicians decided on the overall structure of the song and picked parts of the generated music and lyrics that sounded good together.

"Quantum Trap" is the result of the hard work of our team and I'm really proud of the song we created!

Follow-up

A short while after submitting our song I felt like there were so many interesting improvements to pursue. I read an article about efficient transformers, and how they could be used to process longer sequences.  found a new interesting project by YatingMusic on better learning the structure of songs by grouping words together, called the compound word transformer. This comes including a dataset of 17.000 songs!

I wanted to create something useful that other people who would like to experiment with this technology can use, so I started the "music generation toolbox" repository on Github. The plan is to keep experimenting with new techniques to make the models better and to provide a toolbox for musicians and developers that makes it easy to generate your own music.

You can find the repository including examples on how to use it here:

wingedsheep/music-generation-toolbox

Toolbox for generating music. Contribute to wingedsheep/music-generation-toolbox development by creating an account on GitHub.

GitHubwingedsheep

Some samples of the music generation toolbox:

And some with a perceiver model:

References

Artwork by: Rick Hoving & Joëlle Smidt

AI song contest. (2021). https://www.aisongcontest.com

Music Transformer: Generating Music with Long-Term Structure. (2018, 13 december). Magenta. https://magenta.tensorflow.org/music-transformer

MusicVAE: Creating a palette for musical scores with machine learning. (2018, 15 maart). Magenta. https://magenta.tensorflow.org/music-vae

OpenAI Jukebox. (2020, 4 september). OpenAI. https://openai.com/blog/jukebox/

Payne, C. M. (2020, 19 november). MuseNet. OpenAI. https://openai.com/blog/musenet/

Huang, Yu-Siang, and Yi-Hsuan Yang. "Pop music transformer: Generating music with rhythm and harmony." arXiv preprint arXiv:2002.00212 (2020).

Dai, Zihang, et al. "Transformer-xl: Attentive language models beyond a fixed-length context." arXiv preprint arXiv:1901.02860 (2019).

Aiva Technologies. (2018, 14 september). How we used our Music Engine to create the first AI-generated album of Chinese Music. Medium. https://medium.com/@aivatech/how-we-used-our-music-engine-to-create-the-first-ai-generated-album-of-chinese-music-9d6fa984b4e8

Radford, A. (2021, 4 mei). Better Language Models and Their Implications. OpenAI. https://openai.com/blog/better-language-models/

Brown, Tom B., et al. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

Wang, Ziyu, et al. "Pop909: A pop-song dataset for music arrangement generation." arXiv preprint arXiv:2008.07142 (2020).

Ens, Jeff, and Philippe Pasquier. "Mmm: Exploring conditional multi-track music generation with the transformer." arXiv preprint arXiv:2008.06048 (2020).

Raffel, Colin. Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching. Diss. Columbia University, 2016.

Vaswani, Ashish, et al. "Attention is all you need." arXiv preprint arXiv:1706.03762 (2017).

music21: a Toolkit for Computer-Aided Musicology. (2021). Music21. https://web.mit.edu/music21/

Free Midi - Best Free High Quality Midi Site. (2021). FreeMidi. https://freemidi.org/

Raffel, Colin. (2021). craffel/pretty-midi. GitHub. https://github.com/craffel/pretty-midi

Brockman, G. (2021, 22 juni). OpenAI API. OpenAI. https://openai.com/blog/openai-api/

Hsiao, Wen-Yi, et al. "Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs." arXiv preprint arXiv:2101.02402 (2021).

Wang, P. (2021). lucidrains/reformer-pytorch. GitHub. https://github.com/lucidrains/reformer-pytorch

Wang, P. (2021b). lucidrains/x-transformers. GitHub. https://github.com/lucidrains/x-transformers

Young, K. (2021). kimiyoung/transformer-xl. GitHub. https://github.com/kimiyoung/transformer-xl

View complete python notebook

In Februari 2015 the artificial intelligence company DeepMind published an article in Nature showing they can solve a number of Atari games using Deep Q learning. The amazing part of this is that they didn't create a compact representation of the state of the games. They trained their algorithm purely on the pixels visible on screen and a score. Using this approach they get a better result than all the previous algorithms available, and manage to play 49 out of the 57 games to at least the level of a professional human game tester.

Since reading about their result I have been tinkering around with reinforcement learning. I managed to create some working implementations, but they were never very stable. They learned how to solve a problem, but then after some more training they would oscillate out of control and end up having to learn everything again. An infinite cycle of success and failure. This time I set the challenge for myself to solve the problem, but to also solve it in a way that can maintain its results; A robust deep Q learning algorithm.

OpenAI created a platform for testing reinforcement learning agents. It contains implementations of different problems for reinforcement learning agents to solve, from simple grid worlds to more complex control problems. The goal for this blogpost is to solve the LunarLander-v2 environment.

The goal of lunar lander is to land a small spacecraft between two flags. At every time step you have a choice between 4 actions: fire your main engine, your left engine, your right engine or do nothing. If the spacecraft hits the ground at the wrong angle or too fast it crashes and you get a large penalty. You also get a small penalty for firing your main engine. You get rewards for getting closer to the desired end position, and for touching the ground.

The environment is considered solved if the agent manages to get a score of 200 or more on average in the last 100 episodes.

Q learning

Let's first start by explaining a bit about the algorithm I am going to use. Q-learning is a reinforcement learning algorithm. In reinforcement learning an agent exists inside an environment. At every time step the agent can observe the environment and take an action. After taking an action the environment changes and the agent can observe the new state of the environment. In every transition from one state to another the agent receives feedback in the form of a reward or a penalty.

Q learning is an implementation of a reinforcement learning agent that learns about the environment and tries to maximize the expected reward gained.

It does this by using a Q-function Q(s, a).

Q(s, a) is a function that predicts the future reward in state s when taking action a. If your model is accurately predicting these Q values the agent could pick the action with the highest Q value to maximize its reward.

Usually in Q learning an agent begins by randomly selecting actions and observing their effects in the environment. It then uses the Bellman equation to update the Q function. The Bellman equation looks like this:

The agent was in state s and executed action a, it then ended up in state s' and received reward r. We then update the predicted value Q(s, a) by adjusting the current value with the temporal difference. The temporal difference can be described as the surprise of the reward you received. If the reward you received is higher than expected the temporal difference is positive, and if it is worse than expected the temporal difference is negative. If an action ended up to be better than expected we adjust the Q value upwards, and if it is worse than expected we adjust the Q value downwards.

The values explained in more detail:

• reward: the reward gained by moving from state s to state s' using action a.
• discount factor: how much to take the future into account. If the discount factor is 0 the future is not taken into account at all, if the discount factor is 1 we look an unlimited number of steps into the future. Say for instance the discount factor is 0.99, then we count an action 100 steps into the future only 0.99^100 = 0.366 times of the actual amount.
• learning rate: how much to adjust the Q function in every training step.

Practical example. Say we are in state s and the model predicted Q(s, a) to be 100. We then take action a and end up in state s'. We get a reward of 8. The Q value for the best action we could take in the next state s' is 95. A discount factor of 0.99 is used. We then get a temporal difference of: 8 + 0.99 * 95 - 100 = 2.05. We get 2.05 more reward from the action than expected. We don't want to adjust Q(s, a) to 102.5 right away, because making too large changes to the Q values might cause the model to never converge. So we update the model by the temporal difference multiplied by the learning rate, which is a value between 0 and 1. If the learning rate is 0.1 the new value for Q(s, a) becomes 100 + 0.1 * 2.05 = 100.205. The model is a bit more positive about selecting action a in state s.

The simplest implementation of Q-learning uses a table to hold all the values, but this quickly becomes infeasible for environments with many possible states or actions. By using a (deep) neural network to represent the Q(s, a) function the agent can generalise to never seen combinations of (s, a) better. The input for this neural network is an observation and the output is a Q value for every allowed action in that state. We call this implementation deep Q-learning.

Additions

Deep Q-learning can be quite unstable. Without any tweaks the algorithm can oscillate between good and bad solutions or fail to converge at all.

DeepMind used some additional tricks to make sure their deep Q-learning algorithm converges towards a good solution. In my implementation I am going to implement the following additions.

E-greedy action selection

Reinforcement learning is always a struggle between exploration and exploitation. If you always select the actions you think are the best (exploitation) you might never discover that there are better alternatives. If you explore too much you may never reach the better solutions.

I am going to implement e-greedy action selection. The idea of this method is to have a value between 0 and 1 that determines the amount of exploration vs exploitation. If the value is 0 the agent will always choose the action it thinks is the best one, if the value is 1 it will act completely random. The value starts out high and decreases over time. In the beginning the agent does not know anything about the environment, so it should explore as much to discover the effect of its actions on the environment. As the agent learns more it can start to exploit good strategies.

Experience replay

In the standard Q-learning algorithm the agent learns after every time step, all past experiences are thrown away.

With experience replay we save all state transitions (state, action, reward, new_state) so we can use them for training later.

The advantages of using experience replay are:

• You can use batches of state transitions to learn from. The model gets adjusted for a batch of transitions at once and makes sure that the update is beneficial for predicting each of these transitions. This leads to training updates that are good for multiple state / action pairs at the same time. This means more stable training and better convergence.
• You can revisit past states. Because there is a leaning rate, the model is not instantly adjusted to the correct values. It is nudged a bit in the right direction every training step. If you can revisit interesting actions you can learn more from them.

Double Q-learning

When calculating the target values for updating the model the same model is used for calculating Q(s, a) as for calculating the Q value of the best action in the next state. This can lead to overestimation of the expected reward.

For a detailed explanation read this blogpost by Rubik's code.

To solve this problem we use two models. The main model \(  \theta \)  for selecting actions and a target model \(  \theta ' \)  for calculating the target values. The weights of the main model are copied to the target model every x steps.

Vanilla target: \( R_{t+1} + \gamma max_{a}Q(S_{t+1}, a; \theta) \)
Double q learning target: \( R_{t+1} + \gamma Q(S_{t+1}, argmax_{a} Q(S_{t + 1}, a; \theta); \theta ') \)

\( Q(s, a; \theta) \) means the Q(s, a) value according to model \(  \theta \), so the target is the reward plus the Q value of the target model with the best action in state the next state according to the main model.

Using double q learning leads to more realistic Q-values and thus to better models.

Huber loss

Using mean squared error (mse) as loss function might lead to very high errors when training. These high errors can cause the model to over-adjust and fail to converge to a good solution. Huber loss is less sensitive to outliers because it is exponential only up to a certain threshold and then becomes linear.

Using huber loss leads to a more stable Q-learning convergence.

Implementation

I implemented the algorithm on Google Colab where certain dependencies like Tensorflow come pre-installed. If you want to replicate this code on your local machine you will have to install some more dependencies than I show here.

First we are going to install box2d. This is necessary for running the LunarLander environment.

!pip install box2d

Then we install the necessary modules.

import tensorflow as tf
import gym
import os
import random

from tensorflow.keras.models import Sequential, load_model
from tensorflow.keras.layers import Dense
from tensorflow.keras.regularizers import l2

import numpy as np
import scipy
import uuid
import shutil

import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

import keras.backend as K

We create the LunarLander environment and check how large the state space and the action space are.

The state space describes the number of variables the agent can observe. Since we use observations as input for the Q-learning network this determines the number of inputs our model needs to have.

The action space describes how many actions are possible. For every action we are going to add an output to our Q-learning network.

env = gym.make("LunarLander-v2")

print(f"Input: {env.observation_space}")
print(f"Output: {env.action_space}")

It prints out the following. An observation contains 8 variables and we can choose between 4 different actions.

Input: Box(8,)
Output: Discrete(4)

Then we are going to create our model, but before we can do so we need to implement the Huber loss function. This might be the trickiest part of the implementation.

We implement this as a custom loss function using the Keras backend API. The function takes in all the predicted and true outputs for a batch of state transitions and has to calculate the average Huber loss over this batch. But there is a catch: Our model takes as input a state s, and as output the Q-values for all possible actions. If we want to update the model for a  certain state transition (s, a, r, s') we only want to calculate the loss for the output that belongs to the action a that the agent took to get from state s to s'.

If we have four actions our model could predict [10, 13, 7, 9] as Q-values. In the state transition we are training on we took action 3 and got a target value of 12. We then get a target vector of [0, 0, 12, 0]. The masked_huber_loss function nullifies the error for outputs of which the y_true is 0. So the comparison becomes [0, 0, 7, 0] to [0, 0, 12, 0].

def masked_huber_loss(mask_value, clip_delta):
  def f(y_true, y_pred):
    error = y_true - y_pred
    cond  = K.abs(error) < clip_delta
    mask_true = K.cast(K.not_equal(y_true, mask_value), K.floatx())
    masked_squared_error = 0.5 * K.square(mask_true * (y_true - y_pred))
    linear_loss  = mask_true * (clip_delta * K.abs(error) - 0.5 * (clip_delta ** 2))
    huber_loss = tf.where(cond, masked_squared_error, linear_loss)
    return K.sum(huber_loss) / K.sum(mask_true)
  f.__name__ = 'masked_huber_loss'
  return f

We then create the model as follows. We have 9 input neurons. 8 for the observations in the LunarLander environment and 1 for the fraction complete. Fraction complete indicates how far the epsiode is to ending because of reaching its maximum number of steps. Say we have a maximum of 1000 steps and the current step is 400, then fraction complete is 0.4. This can be used by the agent to decide if it has to act fast, or if it can take it's time.

input_shape = (9,) # 8 variables in the environment + the fraction finished we add ourselves
outputs = 4

def create_model(learning_rate, regularization_factor):
  model = Sequential([
    Dense(64, input_shape=input_shape, activation="relu", kernel_regularizer=l2(regularization_factor)),
    Dense(64, activation="relu", kernel_regularizer=l2(regularization_factor)),
    Dense(64, activation="relu", kernel_regularizer=l2(regularization_factor)),
    Dense(outputs, activation='linear', kernel_regularizer=l2(regularization_factor))
  ])

  optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
  model.compile(optimizer=optimizer, loss=masked_huber_loss(0.0, 1.0))
  
  return model

We can use this model to predict Q-values for a single state or for a batch of states.

def get_q_values(model, state):
  input = state[np.newaxis, ...]
  return model.predict(input)[0]

def get_multiple_q_values(model, states):
  return model.predict(states)

These Q-values can be used to select actions. We use the epsilon greedy implementation like described above.

def select_action_epsilon_greedy(q_values, epsilon):
  random_value = random.uniform(0, 1)
  if random_value < epsilon: 
    return random.randint(0, len(q_values) - 1)
  else:
    return np.argmax(q_values)

def select_best_action(q_values):
  return np.argmax(q_values)

Then we create the replay buffer with a certain size. If the number of state transitions in the replay buffer is bigger than the size it starts overwriting state transitions, starting from the oldest one. The replay buffer has a method to collect a batch of state transitions for training.

class StateTransition():

  def __init__(self, old_state, action, reward, new_state, done):
    self.old_state = old_state
    self.action = action
    self.reward = reward
    self.new_state = new_state
    self.done = done

class ReplayBuffer():
  current_index = 0

  def __init__(self, size = 10000):
    self.size = size
    self.transitions = []

  def add(self, transition):
    if len(self.transitions) < self.size: 
      self.transitions.append(transition)
    else:
      self.transitions[self.current_index] = transition
      self.__increment_current_index()

  def length(self):
    return len(self.transitions)

  def get_batch(self, batch_size):
    return random.sample(self.transitions, batch_size)

  def __increment_current_index(self):
    self.current_index += 1
    if self.current_index >= self.size - 1: 
      self.current_index = 0

When we want to train the model on a batch of state transitions we need to calculate the target values for each of these state / action pairs. We do this using the double Q-learning technique where we use two models for calculating the target values.

def calculate_target_values(model, target_model, state_transitions, discount_factor):
  states = []
  new_states = []
  for transition in state_transitions:
    states.append(transition.old_state)
    new_states.append(transition.new_state)

  new_states = np.array(new_states)

  q_values_new_state = get_multiple_q_values(model, new_states)
  q_values_new_state_target_model = get_multiple_q_values(target_model, new_states)
  
  targets = []
  for index, state_transition in enumerate(state_transitions):
    best_action = select_best_action(q_values_new_state[index])
    best_action_next_state_q_value = q_values_new_state_target_model[index][best_action]
    
    if state_transition.done:
      target_value = state_transition.reward
    else:
      target_value = state_transition.reward + discount_factor * best_action_next_state_q_value

    target_vector = [0] * outputs
    target_vector[state_transition.action] = target_value
    targets.append(target_vector)

  return np.array(targets)

We create a function to train the model using a batch of states and target values.

def train_model(model, states, targets):
  model.fit(states, targets, epochs=1, batch_size=len(targets), verbose=0) 

Then we create a function to make a copy of a model. We need this later to create and update the target model. The model is saved to a backup file, which is loaded into a new object. The copy of the model is returned. We need to pass the masked_huber_loss to the load_model function, because Keras does not know how to initialize the new model otherwise. The backup just has a reference to the loss function name.

def copy_model(model):
  backup_file = 'backup_'+str(uuid.uuid4())
  model.save(backup_file)
  new_model = load_model(backup_file, custom_objects={ 'masked_huber_loss': masked_huber_loss(0.0, 1.0) })
  shutil.rmtree(backup_file)
  return new_model

Since we want to see how well the agent is learning we introduce some logging functionality. The AverageRewardTracker keeps count of the average reward over the last 100 steps. The FileLogger writes the results to a file after every episode.

class AverageRewardTracker():
  current_index = 0

  def __init__(self, num_rewards_for_average=100):
    self.num_rewards_for_average = num_rewards_for_average
    self.last_x_rewards = []

  def add(self, reward):
    if len(self.last_x_rewards) < self.num_rewards_for_average: 
      self.last_x_rewards.append(reward)
    else:
      self.last_x_rewards[self.current_index] = reward
      self.__increment_current_index()

  def __increment_current_index(self):
    self.current_index += 1
    if self.current_index >= self.num_rewards_for_average: 
      self.current_index = 0

  def get_average(self):
    return np.average(self.last_x_rewards)


class FileLogger():

  def __init__(self, file_name='progress.log'):
    self.file_name = file_name
    self.clean_progress_file()

  def log(self, episode, steps, reward, average_reward):
    f = open(self.file_name, 'a+')
    f.write(f"{episode};{steps};{reward};{average_reward}\n")
    f.close()

  def clean_progress_file(self):
    if os.path.exists(self.file_name):
      os.remove(self.file_name)
    f = open(self.file_name, 'a+')
    f.write("episode;steps;reward;average\n")
    f.close()

Then we define the hyperparameters.

replay_buffer_size = 200000
learning_rate = 0.001
regularization_factor = 0.001
training_batch_size = 128
training_start = 256
max_episodes = 10000
max_steps = 1000
target_network_replace_frequency_steps = 1000
model_backup_frequency_episodes = 100
starting_epsilon = 1.0
minimum_epsilon = 0.01
epsilon_decay_factor_per_episode = 0.995
discount_factor = 0.99
train_every_x_steps = 4

Finally it all comes together in the main loop.

replay_buffer = ReplayBuffer(replay_buffer_size)
model = create_model(learning_rate, regularization_factor)
target_model = copy_model(model)
epsilon = starting_epsilon
step_count = 0
average_reward_tracker = AverageRewardTracker(100)
file_logger = FileLogger()

for episode in range(max_episodes):
  print(f"Starting episode {episode} with epsilon {epsilon}")

  episode_reward = 0
  state = env.reset()
  fraction_finished = 0.0
  state = np.append(state, fraction_finished)

  first_q_values = get_q_values(model, state)
  print(f"Q values: {first_q_values}")
  print(f"Max Q: {max(first_q_values)}")

  for step in range(1, max_steps + 1):
    step_count += 1
    q_values = get_q_values(model, state)
    action = select_action_epsilon_greedy(q_values, epsilon)
    new_state, reward, done, info = env.step(action)
    
    fraction_finished = (step + 1) / max_steps
    new_state = np.append(new_state, fraction_finished)
    
    episode_reward += reward

    if step == max_steps:
      print(f"Episode reached the maximum number of steps. {max_steps}")
      done = True

    state_transition = StateTransition(state, action, reward, new_state, done)
    replay_buffer.add(state_transition)

    state = new_state

    if step_count % target_network_replace_frequency_steps == 0:
      print("Updating target model")
      target_model = copy_model(model)

    if replay_buffer.length() >= training_start and step_count % train_every_x_steps == 0:
      batch = replay_buffer.get_batch(batch_size=training_batch_size)
      targets = calculate_target_values(model, target_model, batch, discount_factor)
      states = np.array([state_transition.old_state for state_transition in batch])
      train_model(model, states, targets)

    if done:
      break

  average_reward_tracker.add(episode_reward)
  average = average_reward_tracker.get_average()

  print(
    f"episode {episode} finished in {step} steps with reward {episode_reward}. "
    f"Average reward over last 100: {average}")

  if episode != 0 and episode % model_backup_frequency_episodes == 0:
    backup_file = f"model_{episode}.h5"
    print(f"Backing up model to {backup_file}")
    model.save(backup_file)

  epsilon *= epsilon_decay_factor_per_episode
  epsilon = max(minimum_epsilon, epsilon)

When the training is done we can use the following code to visualise the training progress by loading the log file and plotting it using matplotlib.

data = pd.read_csv(file_logger.file_name, sep=';')

plt.figure(figsize=(20,10))
plt.plot(data['average'])
plt.plot(data['reward'])
plt.title('Reward')
plt.ylabel('Reward')
plt.xlabel('Episode')
plt.legend(['Average reward', 'Reward'], loc='upper right')
plt.show()

Results

After training the model for 578 steps I stopped training the model, because the model stopped improving. The average score over 100 episodes at that point was 248.97; a lot more than the required score of 200.0 to beat the environment.

Now it was time to see the model in action. To do this I loaded the last version of the model, turned off the training and rendered the results to the screen. It is actually working!

Although the model manages to get a pretty good score I still think it is quite slow at landing. In a future iteration of the algorithm I might add a penalty for taking too long.

Notes

2021-10-14: How to reproduce the visualisations?

I was asked a question about how to create the visualisations, and if they can be done in Colab. Here is my answer:

I didn't do the rendering on Google Colab, but on my local machine, because the OpenAI gym environment renders it in a window and this does not work in Colab.

Before you can run the script locally you need to have python and pip installed, and you need to install the following packages using "pip" https://pypi.org/project/pip/.

* gym: https://pypi.org/project/gym/
* tensorflow: https://pypi.org/project/tensorflow/
* numpy: https://pypi.org/project/numpy/
* box2d: https://pypi.org/project/box2d/

Here is a script that you can run locally to visualise runs of a certain model:

import gym
from tensorflow.keras.models import load_model
from tensorflow.keras.losses import MeanSquaredError
import numpy as np

env = gym.make("LunarLander-v2")
filename = "model_xxx.h5"
trained_model = load_model(filename, custom_objects={"masked_huber_loss": MeanSquaredError()})

evaluation_max_episodes = 10
evaluation_max_steps = 1000


def get_q_values(model, state):
    input = state[np.newaxis, ...]
    return model.predict(input)[0]


def select_best_action(q_values):
    return np.argmax(q_values)

rewards = []
for episode in range(1, evaluation_max_episodes + 1):
    state = env.reset()

    fraction_finished = 0.0
    state = np.append(state, fraction_finished)

    episode_reward = 0

    step = 1
    for step in range(1, evaluation_max_steps + 1):
        env.render()
        q_values = get_q_values(trained_model, state)
        action = select_best_action(q_values)
        new_state, reward, done, info = env.step(action)

        fraction_finished = (step + 1) / evaluation_max_steps
        new_state = np.append(new_state, fraction_finished)

        episode_reward += reward

        if step == evaluation_max_steps:
            print(f"Episode reached the maximum number of steps. {evaluation_max_steps}")
            done = True

        state = new_state

        if done:
            break

    print(f"episode {episode} finished in {step} steps with reward {episode_reward}.")
    rewards.append(episode_reward)

print("Average reward: " + np.average(rewards))

In the demo there is a line that says

filename = "model_xxx.h5"

Here you can enter the name of the model you downloaded. If you look at the output of the Colab script you can see the models that are saved on the checkpoints. I updated the code in the blog to save the models as ".h5", which is the keras format for saving models. The advantage of this is that it is a single file that you can download from your Colab environment.

If you download a model you can put it in the same folder as the "lunarlander_demo.py" script, adjust the filename to point to and run the script.The environments of OpenAI have a "render()" function which renders the current state of the environment to the screen. This is used to render the runs.