CADENCE: Teaching a Music Model to Think in Form, Not Just Notes

Abstract

Today’s symbolic-music models can produce convincing passages and still fail at the thing listeners notice over time: intention. A phrase begins, a texture develops, and then the music forgets what it was becoming. CADENCE—Causal Architecture for Dual-rate Expressive Note Continuation and Evolution—is my attempt to design around that failure. It separates composition into two synchronized timescales: a low-rate planner that represents bars, phrases, harmony, motifs, and formal direction, and a high-rate renderer that realizes those decisions as expressive notes, timing, dynamics, and pedal. Both stages combine recurrent memory with selective attention, while a compact compound-event tokenizer preserves performance detail in roughly one renderer step per note. The result is a research-backed architecture specification and a runnable 11.6-million-parameter continuation alpha, not yet a claim of state-of-the-art musical quality. This article explains the problem, the design, the evidence behind it, the unresolved risks, and the experiments required to determine whether CADENCE can turn local fluency into long-range musical thought.

The problem is not note prediction

Music generation is often described as next-token prediction, but music is experienced as nested time. A pianist shapes milliseconds of attack and release; a motif occupies seconds; a phrase unfolds across bars; a section earns its return over minutes. The same piece must therefore be coherent at several temporal scales at once. A model can predict an individually plausible note without knowing why the phrase exists, where the harmony is heading, or whether a previous idea should return transformed rather than copied.

The historical case for long-range modeling is strong. Music Transformer showed that relative attention could produce minute-long piano sequences and coherent motif continuations, explicitly motivating self-attention through music’s dependence on repetition and relative timing.¹ REMI then made beat and bar structure explicit, improving the rhythmic organization of generated piano music rather than asking a network to infer every metrical relationship from a flat stream.² Later systems pursued controllable descriptions, hierarchical summaries, and more efficient compound events.³⁴⁵

Yet there is a persistent engineering tradeoff. Full attention offers direct retrieval: a model can compare the current passage with a motif hundreds or thousands of events earlier. Its memory and key-value cache grow with the sequence. Recurrent and state-space models offer cheaper streaming, but compress the past into fixed-size state. Mamba demonstrated that selective state-space models can achieve linear sequence scaling,⁶ while Griffin and RecurrentGemma showed the practical viability of hybrid recurrence and local attention.⁷ Neither result implies that recurrence alone is the right memory primitive for music. A sonata does not merely need a vague trace of its opening; it may need exact interval relations, a transposed motif, and a meaningful formal return.

CADENCE begins from a simple position: local fluency and global intention are related, but they are not identical jobs. A system should not spend the same computational currency on a four-millisecond timing adjustment and a four-bar phrase decision.

One model, two clocks

CADENCE separates symbolic piano modeling into a structural planner and an event renderer. The planner operates at beat and bar rate. It represents meter, tempo, harmonic evidence, rhythmic density, register, phrase boundaries, section roles, motif identity, and repeat or variation pointers. The renderer operates at event rate. It consumes the plan and emits the performance: onset timing, pitch, duration, velocity, chord voicing, and pedal changes.

This is not a return to rigid rule-based composition. The planner is generative, its descriptors may be sampled or edited, and uncertain analytical labels can be masked or downweighted. It is better understood as an explicit workspace between a control prompt and the final performance. FIGARO provides evidence that high-level descriptions can support fine-grained control,³ while Museformer shows the usefulness of combining fine attention to relevant bars with coarse bar summaries for long-form generation.⁴ CADENCE joins those ideas to an expressive performance stream.

The distinction matters because planning and rendering have different entropy. The number of plausible surface realizations of a phrase is enormous: timing can breathe, voices can redistribute, velocities can curve, and pedals can overlap. A bar-level structural decision is lower-rate and more inspectable. CADENCE therefore allocates most network depth to the renderer while giving the planner a long musical horizon at relatively low cost.

The central bet is that a model should decide what a passage is doing before it decides every detail of how the passage is played.

The architecture: persistent state plus exact retrieval

Both planner and renderer use the same basic pre-normalized residual block: a sequence mixer followed by a SwiGLU feed-forward layer. The sequence mixer alternates between Gated DeltaNet recurrence and sparse attention, at roughly three recurrent blocks for every attention block.

Gated DeltaNet combines gated forgetting with a delta-rule associative update. Its published results are primarily in language, and its strongest reported hybrid retains sliding-window attention.⁸ That makes it useful evidence for efficient persistent memory, not proof of musical superiority. In CADENCE, recurrent blocks carry local continuity, active state, and inexpensive history. Attention remains available where music benefits from exact comparison.

The renderer’s sparse attention has several routes:

• causal local attention over recent performance events;
• global access to structural-plan tokens and current bar or section summaries;
• coarse access to earlier bar summaries;
• fine access to musically relevant earlier bars selected by motif pointers and fixed lags;
• relative features for event distance, elapsed musical time, within-bar phase, and pitch interval.

The architecture is deliberately hybrid. Pure full attention is direct but expensive at long event contexts. Pure recurrence is efficient but makes exact retrieval indirect. A planner without an expressive renderer risks producing competent outlines with lifeless performances; a renderer without a planner risks producing beautiful surfaces without form. CADENCE assigns each mechanism to the relationship it represents best.

Two task-specialized checkpoints

Each model size has two specialist weight sets. CADENCE-G generates from scratch, learning authentic openings, complete plans, and endings. CADENCE-C continues a supplied performance, analyzing only the observed seed, generating a future plan, and receiving extra loss weight near the join. They share the architecture and representation, so base pretraining, data infrastructure, and evaluation tools can be reused without pretending that a convincing opening and a faithful continuation are the same conditional distribution.

Representation is part of the model

A sequence model never sees “music” in the abstract. It sees a representation, and that representation decides which relationships are easy, which sequences are long, and which errors are even possible. A NoteOn/NoteOff stream is expressive but verbose and stateful. A rigid piano roll exposes metrical structure but can erase timing nuance. A large Cartesian vocabulary becomes sparse. CADENCE-EPC—the project’s Expressive Performance Compound tokenizer—uses a synchronized dual stream to avoid forcing one compromise onto both form and performance.

In the renderer stream, elapsed time is quantized to four milliseconds. A normal note is represented by one ragged compound record containing mixed-radix delta onset, pitch, explicit duration, exact coarse-to-fine velocity, and relevant grouping information. Sparse first-class events represent sustain, sostenuto, and soft-pedal changes. Near-simultaneous notes become onset groups ordered from the highest pitch downward, so a chord can store an absolute top note followed by descending intervals. The planner independently carries reliable beat and bar coordinates and structural descriptors.

The compound design follows a well-supported efficiency argument. Compound Word Transformer grouped neighboring typed attributes, shortened sequences, and reported five-to-ten-times faster training convergence than its baselines with comparable generation quality.⁵ But the fields inside a note are not independent: pitch changes the plausible duration, dynamics, and voicing. Nested Music Transformer addresses this by decoding compound attributes sequentially.⁹ CADENCE specifies a time-shifted delay pipeline so dependent fields remain causal without requiring a full serial subdecoder call for every note.¹⁰

The renderer uses 640 embedding rows across its factor tables; the fixed planner fields use approximately 621 more. This is small compared with a flat vocabulary containing every possible pitch-duration-velocity-time combination. The decoder also applies a hard grammar: chord pitches descend and stay in range, event families expose only legal fields, elapsed time never reverses, releases apply only to open notes, and end-of-sequence cannot silently strand key-down notes. Grammar guarantees syntactic MIDI validity, not musical taste.

There is also a context-accounting consequence. A compound renderer event is roughly one step per note, plus sparse controls. A flat baseline that uses multiple symbols per note cannot be compared fairly at the same raw token count. Musical span should be matched by duration, bars, or approximate notes. Otherwise, representation efficiency can be mistaken for architectural intelligence.

Continuation without peeking into the future

Continuation looks simple until the cut lands under a held note. Suppose a pianist strikes middle C before the boundary and releases it afterward. If the observed prefix stores that note’s final duration, it has leaked the future. Sustain pedal creates the same problem: the physical key may be up while the note is still sounding, and a future pedal release determines when the sound ends.

CADENCE treats the join as a causal state-transfer protocol:

1. Any key-down note crossing the cut has its duration replaced with OPEN.
2. A BOUNDARY record snapshots the clock, structural position, confidence, and all supported pedal values.
3. Zero-time STATE records describe each sounding pitch, its onset age, attack velocity, and whether it is physically held or pedal-sustained.
4. The future model generates a RELEASE for an open key-down note or a later pedal change for a pedal-sustained note.
5. All observed-prefix targets that could reveal a post-boundary release or pedal change are masked.

This protocol is one of the project’s most concrete contributions because it turns an informal prompting operation into a testable causal contract. The same transform must be applied to training crops and user prompts. Validation should perturb the unseen suffix and prove that the encoded prefix does not change. Round-trip tests alone are insufficient: a codec can decode correctly and still leak target information.

A family from 12M to 748M parameters

CADENCE is specified as a model family rather than a single oversized experiment. Parameter figures below are estimator-level design counts; the continuation alpha uses a different portable backend and is discussed separately.

The depth allocation encodes the project’s thesis: scaling primarily buys more performance realization capacity, while the planner grows slowly because it operates at low rate. The smallest configuration is meant to validate mechanics. The 40M model is a practical data-system target. The 85M model is large enough for a serious listening decision. The 748M configuration is an Orpheus-class research target, not a sensible starting point.

The open Orpheus model provides a useful reference: its model card describes a 748M autoregressive Transformer, an 8,192-token context, and training on 2.31 million selected Godzilla samples.¹¹ CADENCE does not treat those headline numbers as a quality guarantee. Context lengths depend on tokenization, and the Orpheus inference guidance itself emphasizes seeded continuation. The right comparison is equal musical span, equal candidate budget, and blinded listening—not an isolated parameter count.

Data, provenance, and the cost of convenience

A careful architecture can still be defeated by an incoherent corpus. MIDI collections mix native performances, score exports, multitrack arrangements, audio transcriptions, duplicates, repairs, and files with uncertain rights. CADENCE therefore proposes one canonical parser and one versioned tokenizer applied to permissible raw MIDI, with transformations and provenance stored beside every sequence.

The training mixture is divided into three quality tiers:

• Gold: aligned or verified expressive piano performances, led by MAESTRO and similarly trustworthy sources.
• Silver: deduplicated piano-oriented corpora such as Aria-MIDI and GiantMIDI-Piano, plus carefully filtered native piano from larger collections.
• Bronze: uncertain transcriptions and projected multitrack material, retained for diversity but downweighted.

MAESTRO 3.0.0 contains roughly 199 hours of aligned audio and MIDI and provides composition-level train, validation, and test splits.¹² GiantMIDI-Piano reports 10,855 works, 38.7 million notes, and 2,786 composers, largely obtained through piano transcription from online audio.¹³ Aria-MIDI contributes a million-file piano corpus and explicitly provides pruned and deduplicated subsets.¹⁴ Lakh contains 176,581 deduplicated MIDI files but warns that copyright status is difficult to determine from the source material.¹⁵ These resources are valuable for different reasons; their file counts are not interchangeable measures of musical fidelity.

Deduplication must happen across the union of corpora before splitting. Exact hashes catch identical files, but musical leakage also requires normalized note fingerprints, tempo- and transposition-invariant shingles, similarity retrieval, and grouping by work or performance family. A random clip split can place two arrangements of the same composition on opposite sides of the evaluation boundary and make memorization look like generalization.

The legal boundary is equally important. Dataset-level licenses do not necessarily clear every underlying composition, performance, or transcription. Several large public MIDI corpora are restricted to noncommercial research. CADENCE’s proposed mixture should therefore be treated as a research route; commercial deployment requires a separate provenance-complete corpus and legal review.

Training curriculum

Shared pretraining mixes full-piece next-event modeling, suffix continuation, and structural-plan reconstruction. Auxiliary objectives teach the renderer to agree with planned bar descriptors and teach repeat pointers to retrieve related motifs. Long-context training then increases the share of complete pieces and high-quality data before the model forks into generation and continuation specialists. A conservative gold-data finish is intended to improve expressive fidelity without erasing the diversity learned earlier.

Approximate budgeting targets range from 0.3 billion compound renderer steps for the 12M model to 15–20 billion for the 748M model. These are engineering priors, not scaling-law claims. The real decision boundary should be the point at which additional compute produces reliable gains in blinded musical judgment.

How the idea should be judged

Symbolic generation is especially vulnerable to convenient metrics. Perplexity helps debug optimization, but it is representation-dependent and does not directly measure whether a piece develops, returns, or ends. The evaluation must therefore ask whether the complete system works for its intended musical tasks.

For continuation, prompts should end at bar lines, mid-bar, under held sustain, immediately before motif returns, and across tempo or meter changes. The first generated events should be checked for active-note and pedal correctness. The first two bars deserve a separate listener judgment because a model can recover from a bad seam and still receive an acceptable whole-sample score.

For full generation, fixed 16-, 32-, and 64-bar pieces should be assessed for openings, phrase development, contrast, section recurrence, climax, and ending. Bar-level self-similarity, motif retrieval under transposition, harmonic tension, cadence placement, density, velocity, and pedal statistics are useful diagnostics. None is a complete quality oracle.

Final comparisons should use blinded paired listening or MUSHRA-style panels with the same piano renderer, prompt, duration constraints, and sampling budget. Results should separate musicians from general listeners, report confidence intervals and rater reliability, and compare against held-out human performances as well as model baselines. Generated samples also need exact and invariant nearest-neighbor retrieval against training data to estimate memorization.

This is an important philosophical constraint: CADENCE should not win because its diagrams are persuasive. It should win only if listeners prefer its continuations and long-form pieces under fair, predeclared conditions.

What exists, and what remains speculative

The CADENCE repository contains the architecture specification, tokenizer contract, validation tests, model configurations, data and evaluation plans, parameter-estimation tools, and a continuation-first implementation. The current portable alpha instantiates at exactly 11,611,840 parameters and includes an end-to-end notebook for dataset auditing, streaming tokenization, one-pass training, checkpointing, and MIDI smoke generation. It validates renderer mechanics and causal boundary semantics. The full structural planner and the proposed Gated DeltaNet-plus-sparse-attention backbone remain future implementation milestones.

That distinction is crucial. No published paper validates the exact CADENCE system as a whole. The four-millisecond clock, compound grouping rules, explicit boundary records, planner schema, sparse retrieval routes, and scale allocation are engineering inferences assembled from adjacent evidence. Gated DeltaNet has not yet been established at scale for symbolic music. Beat, chord, phrase, and motif labels can be noisy. Sparse attention may prove redundant at short contexts or indispensable only at long ones. A simpler Transformer may still sound better.

There are also implementation gates before scale. The codec should demonstrate deterministic and idempotent round trips, exact pitch and velocity preservation after canonical repair, bounded timing error, zero grammar failures, no continuation-prefix leakage, and its expected compression. The 12M system should establish mechanics; the 40M system should establish the data pipeline; the first powered listening decision should happen around 85M. A 748M run is justified only after quality, throughput, provenance, licensing, and memorization gates are met.

CADENCE is therefore best understood as a falsifiable design thesis. It argues that musical intelligence benefits from an explicit distinction between intention and realization, and that efficient persistent memory should coexist with exact musical retrieval. The project’s value is not that every choice must survive. Its value is that the choices are specific enough to implement, inspect, test, and reject.

Conclusion: from plausible notes to musical consequences

The hardest part of generative music is not producing the next acceptable note. It is making that note feel like a consequence of what came before and a promise about what comes next. CADENCE approaches that problem with two clocks: one slow enough to reason about form, and one fast enough to preserve performance. Its planner makes structural intent visible. Its renderer protects timing, dynamics, and pedal. Its hybrid memory treats recurrence and attention as complementary tools. Its continuation protocol refuses to cheat at the boundary.

Whether this architecture produces better music is an empirical question, and the honest answer does not yet exist. But the design changes the question from “How large a next-token model can we train?” to “What kinds of memory and representation does musical thought require?” That is the experiment CADENCE is built to run.

References and further reading

1. Huang, C.-Z. A., et al. (2018). Music Transformer. arXiv:1809.04281.
2. Huang, Y.-S., & Yang, Y.-H. (2020). Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions. ACM Multimedia 2020.
3. von Rütte, D., Biggio, L., Kilcher, Y., & Hofmann, T. (2022). FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control. arXiv:2201.10936.
4. Yu, B., Lu, P., Wang, R., Hu, W., Tan, X., Ye, W., Zhang, S., Qin, T., & Liu, T.-Y. (2022). Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation. NeurIPS 2022.
5. Hsiao, W.-Y., Liu, J.-Y., Yeh, Y.-C., & Yang, Y.-H. (2021). Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs. AAAI 2021.
6. Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.
7. De, S., et al. (2024). Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models; Botev, A., et al. (2024). RecurrentGemma: Moving Past Transformers for Efficient Open Language Models.
8. Yang, S., Kautz, J., & Hatamizadeh, A. (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv:2412.06464.
9. Yoo, H., Dong, H.-W., Jung, J., & Jeong, D. (2024). Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generation. ISMIR 2024.
10. Wang, T.-K., Tan, C.-P., & Yang, Y.-H. (2025). Time-Shifted Token Scheduling for Symbolic Music Generation. arXiv:2509.23749.
11. Project Los Angeles. Orpheus Music Transformer model card and repository. Hugging Face.
12. Hawthorne, C., et al. (2018); Google Magenta. MAESTRO Dataset, version 3.0.0.
13. Kong, Q., et al. (2020). GiantMIDI-Piano: A Large-Scale MIDI Dataset for Classical Piano Music. arXiv:2010.07061.
14. Bradshaw, L., & Colton, S. (2025). Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling; official repository.
15. Raffel, C. (2016). The Lakh MIDI Dataset.
16. Oore, S., Simon, I., Dieleman, S., Eck, D., & Simonyan, K. (2018). This Time with Feeling: Learning Expressive Musical Performance. Neural Computing and Applications.
17. Fradet, N., Briot, J.-P., Chhel, F., El Fallah Seghrouchni, A., & Gutowski, N. (2023). Impact of Time and Note Duration Tokenizations on Deep Learning Symbolic Music Modeling. ISMIR 2023.
18. Zeng, M., Tan, X., Wang, R., Ju, Z., Qin, T., & Liu, T.-Y. (2021). MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training. Findings of ACL-IJCNLP 2021.
19. Fradet, N., Gutowski, N., Chhel, F., & Briot, J.-P. (2023). MidiTok: A Python Package for MIDI File Tokenization. ISMIR 2023 Late-Breaking Demo.
20. Li, S., et al. (2023). Byte Pair Encoding for Symbolic Music. arXiv:2301.11975.
21. Li, S., et al. (2024). An Analysis of Byte Pair Encoding for Symbolic Music. arXiv:2410.01448.
22. MetaCreate. (2025). GigaMIDI: A Large-Scale MIDI Dataset with Musical Metadata; dataset card.
23. Project Los Angeles. Discover MIDI Dataset; Godzilla MIDI Dataset. Hugging Face.