An NLP Audit of a Cultural Narrative: Pre-Registration, Cluster Permutation, and Placebo Calibration on Drake's Lyrics

Pre-registration frozen 2026-05-20 00:26 UTC, before any analysis ran. Single protocol deviation logged: a Genius-HTML coverage augment to fill in 47 missing 2024–2026 tracks. Surface metrics (§4) are confirmatory; embedding, within-genre, cluster-permutation, placebo, and held-out-album analyses (§5, §7, §8, §9) are exploratory. All numbers reproducible from §13.

TL;DR§

Question. Did Drake’s lyrics measurably change after the 2024 Kendrick beef, in the directions the discourse insists they did?

Pipeline. We analyzed Drake’s May 2026 triple drop — Iceman, Habibti, and Maid of Honour — against the rest of his discography (n = 338 songs, 50 albums, 2009 → 2026) using spaCy POS/NER surface metrics, all-MiniLM-L6-v2 sentence embeddings, a distilroberta emotion classifier, TF-IDF logistic regression, and segmented OLS with HAC standard errors. All four pre-registered surface predictions returned null. The embedding shift that looked real at the song level (p ≈ 0.0002) collapsed once permutation respected the obvious album-cluster structure (p = 0.11), agreed across two encoders, sat in the bottom quartile against 23 placebo time-cuts, and failed to appear in a peer artist’s catalog at the same cutoff.

Findings.

1. Pre-registered surface metrics (confirmatory): 0 / 4 pass Bonferroni at α = 0.0125. Proper-noun density, I/(I+we), defensive deixis, Toronto anchoring all null. The cultural narrative’s specific surface-level claims fail.
2. Embedding centroid drift (exploratory, song-level permutation): L2 distance pre vs post = 0.158 vs null mean 0.097, permutation p = 0.0002. A semantic-level shift if individual songs are the inferential unit.
3. Interpretability check: TF-IDF logistic regression hits 5-fold CV AUC = 0.76 — well above chance but not overwhelming. Top features and centroid-axis projection blame the May 2026 R&B-leaning triple drop (Habibti, Maid of Honour) as a genre-mix confounder.
4. Within-genre re-test (exploratory, song-level permutation): Hand-tagging albums into rap / rnb_pop / mixed, the rap-only test still shifts: p = 0.0034. The genre-confounder reading was too strong.
5. Album-level cluster permutation (exploratory robustness check): When permutation moves from songs to whole albums — the actual unit of release — every embedding p-value drops back above 0.05. Unconditional: p = 0.11. Within-rap: p = 0.35. rap_or_mixed: p = 0.07. Exclude triple drop: p = 0.07. The exploratory “shift” was carried in part by pseudo-replication inside album clusters.
6. Sliding-cutoff sensitivity (±6 months): Re-running the cluster test at the event date ± 6 months. Cluster p ranges from 0.058 (six months early, when Drake’s own diss tracks move into the post window) to 0.222 (six months late). No cut anywhere in the window crosses α = 0.05. L2 stays within 0.150–0.161 — the verdict isn’t an artifact of one arbitrary date choice.
7. Placebo time-cut calibration: Re-running the cluster test at 23 pre-committed non-event candidate dates (every 6 months, 2012-07-01 → 2023-07-01). The May 2024 cut produces a smaller L2 than 19 of 23 placebos (83% of placebos are more shifted) and a larger cluster p than all 23. Even placebos with the same album imbalance produce L2 ≈ 0.22 vs. 0.158 at May 2024. The May 2024 result is less unusual than a random temporal split would be.
8. Peer-artist control (J. Cole): Same pipeline, same 2024-05-04 cut on J. Cole’s catalog (n = 197, 19 pre / 5 post albums). L2 = 0.110, cluster p = 0.759. The Drake null is not Drake-specific — the closest clean peer artist’s catalog also fails to show a May 2024 shift, and his L2 distance is smaller than Drake’s. (Lil Wayne and Future were attempted but their post-event Deezer coverage was dominated by catalog re-releases / a single mixtape; see §8.)
9. Held-out album train/test (exploratory model-side robustness): Leave-one-album-out over 9 post + 6 pre-event albums with TF-IDF and MiniLM features. Pooled LOAO AUC = 0.63 (both feature sets, bootstrap 95% CI [0.54, 0.73]) — vs the song-level CV’s 0.76. The May 2026 triple drop classifies as pre-Kendrick under both representations: Iceman P(post) = 0.37 / 0.39, Habibti 0.47 / 0.42, Maid of Honour 0.50 / 0.42. A classifier trained on the rest of Drake’s catalog — including 6 other post-event releases — cannot recognize his biggest post-event project as post-event.

Takeaway, as an NLP researcher would write it. Two claims survive cleanly. First, the four most-cited folk mechanisms — receipts, isolation, defensive deixis, Toronto-anchoring — are confidently falsified. All directional, all pre-registered, all null. Second, the holistic semantic shift the song-level embedding test seemed to find does not survive any of the appropriate robustness checks. Under album-level permutation every embedding p-value rises above 0.05. Against 23 placebo time-cuts, the May 2024 result is less shifted than a typical random split — 19 of 23 placebos produce larger L2 than the real cut, and all 23 produce smaller cluster p-values. The same pipeline applied to a peer artist (J. Cole) at the same May 2024 cutoff returns a firmer null (p = 0.76, L2 = 0.110 < Drake’s 0.158). The original p = 0.0002 song-level signal was not merely under-powered once corrected — it was consistent with “no Kendrick-specific shift beyond normal year-to-year drift.” The methodological lesson: with low post-event sample sizes, song-level permutation can manufacture orders of magnitude of apparent evidence. Cluster-level inference, placebo time-cut calibration, and a peer-artist control are what determine whether the signal is real.

Read: the four discourse-named surface mechanisms are confidently falsified. The one apparent semantic shift (tests 2 & 4) appears only when songs are treated as iid — under album-cluster permutation, a sliding cutoff, 23 placebo cut dates, a peer artist at the same cut, and held-out album classification, the same data returns null. SHIFT * = significant under song-level permutation only; does not survive any test in §8 or §9.

1. The premise§

The topic happens to be Drake and hip-hop. The actual question is a computational one: given a small, clustered text corpus and a cultural narrative claiming a behavioral change at a specific date, what does a defensible NLP pipeline look like, and what does it conclude? That’s the part worth carrying forward. Drake is a convenient test bed because the discourse around him gives unusually specific surface-level predictions (more receipts, more isolation, more defensive deixis, more Toronto-anchoring) that a competent NLP pipeline can pre-register and falsify directionally.

The substantive premise: on 2024-05-04 Kendrick Lamar released Not Like Us. The cultural consensus is that it ended Drake’s run as the consensus pop-rap apex, and that Drake has not been the same artist since. The first claim is about chart position and is well-documented. The second is about Drake himself — his work, his stance, his voice — and is mostly vibes.

This is a one-evening study to ask the second claim quantitatively. If Drake changed after May 2024 in the directions the discourse insists he changed, the change should be visible in his words. The CS-side challenges that make this a useful worked example: the post-event corpus is small (9 distinct release events), songs cluster inside albums so observations are not iid, the cultural prior is strong enough to motivate honest pre-registration, and the embedding analysis only becomes interpretable once the inferential unit is stress-tested and calibrated against placebo time-cuts. The hip-hop subject is incidental; the same pipeline shape would apply to “did this politician’s rhetoric shift after the scandal” or “did the company’s earnings calls change tone after the lawsuit” — any text corpus with a small post-event window and album-like clustering.

2. What I pre-registered (before any data)§

Four predictions, all directional, all on monthly-aggregated medians:

Design: interrupted time series, y_t = β₀ + β₁·t + β₂·D_t + β₃·P_t + ε. D is post-event indicator (level shift); P is months-since-event (slope change). β₂ is the primary coefficient. HAC standard errors (maxlags=3). Bonferroni across four predictions: α-per-test = 0.0125.

Decision rule, also set in advance: promote to paper if ≥ 2 of 4 predictions pass Bonferroni and a third is in the predicted direction at p < 0.05 unadjusted. Anything less ⇒ blog post.

3. The data§

408 Drake-primary releases enumerated via the Deezer public artist endpoint (artist 246791) covering 2009-07-02 → 2026-05-15, including the May 15 2026 triple drop (Iceman, Habibti, Maid of Honour). Lyrics from lyrics.ovh where indexed; Genius song-page HTML (no auth required for the public pages) for the rest. Filters: Drake as primary, parseable release date, ≥ 80 lyric tokens after boilerplate stripping. Final n = 338 (283 pre, 55 post, distributed across 42 monthly bins — 36 pre, 6 post).

One important data-cleaning detail that mattered downstream: lyrics.ovh returns plain lyrics with section markers ([Verse 1: Drake], [Chorus]) already stripped. Genius’s HTML keeps them and prepends page-level boilerplate (“176 Contributors / Translations / Português / Russian / It’s Up Lyrics / Read More”). The first pass through the embedding analysis turned up “lyrics”, “intro”, “chorus”, “russian”, “translations” as top post-event features — a pure data-source artifact, not Drake. A second cleaning pass (clean_jsonl.py) strips both the page boilerplate and the bracket markers from all records before any downstream analysis. The findings below survive that pass. The earlier raw-source version is preserved at data/drake.raw.jsonl for audit.

4. The pre-registered confirmatory test: null on all four§

0 of 4 pass Bonferroni. Only 1 of 4 is even in the predicted direction. The descriptive pre/post means agree — tiny shifts indistinguishable from drift. Per the frozen decision rule, this is a blog post, not a paper.

One coefficient does survive Bonferroni: P1’s β₃ (slope change) = −0.00080 per month, p = 5.99 × 10⁻⁵. Drake’s proper-noun density had been rising steadily for fifteen years — a slow β₁ of receipts, names, and cities compounding album over album. Then May 2024 happened and the slope flipped. Not the level — the trajectory. By my own protocol β₃ is secondary, so this gets called “suggestive” rather than confirmatory, and the direction also runs opposite to P1’s prediction. Pre-registration is the only reason I’m calling it suggestive rather than spinning it into the headline.

5. The exploratory follow-up: an embedding shift at the song level§

The four surface metrics are coarse. A reasonable critique is that they couldn’t detect a persona shift even if one existed. So a follow-up using a sentence-transformer should be more sensitive.

I embedded every Drake song with all-MiniLM-L6-v2 (384-dim), L2-normalized each song vector, then computed the L2 distance between the pre-event and post-event centroids. A permutation test with 5,000 random pre/post relabelings gives the null distribution of centroid distances under no real shift.

Three more exploratory probes:

The emotion classifier sees no shift in the 7-emotion mix. Lexical diversity is flat. The NER breakdown shows that the propn signal from §4 is not coming from PERSON or GPE — both essentially identical pre/post; if anything, ORG (companies, labels, brands) drifted up.

So we have one apparently strong exploratory signal (the song-level embedding shift) and four null ones (emotion, TTR, MTLD, NER-by-type). Something is moving in the embedding that the categorical probes can’t see — at least under the assumption that individual songs are exchangeable observations. §8 stress-tests that assumption.

6. The interpretability check — and the confounder§

To understand what is driving the embedding shift, I trained a TF-IDF logistic-regression classifier (1- and 2-grams, min-df 4) to predict pre vs post from raw lyrics. 5-fold CV AUC was 0.76 — well above the 0.5 chance line but not overwhelming; the discriminative signal exists but is not so strong that the classifier is memorizing per-song giveaways. The top coefficient features per class:

Top terms pointing POST-event (2024–2026): ass · stick · waist · lit · iceman · 6ix · pussies · ride · eye · trips · act like · like yeah · ayy yeah

Top terms pointing PRE-event (2009–2024): oh · nigga · money · late · young · night · change · things · city · days · dawg · kid · road · treated · knew · trying · somebody · think

And the top-10 most “post-Kendrick-sounding” tracks by projection onto the centroid axis:

• NOKIA ($ome$exy $ongs 4 U, 2025)
• Classic (Habibti, 2026)
• Where's Your Stuff Interlude (Maid of Honour, 2026)
• Cheetah Print (Maid of Honour, 2026)
• B's On The Table (Iceman, 2026)
• Housekeeping Knows (100 GIGS, 2024)
• Rusty Intro (Habibti, 2026)
• Outside Tweaking (Maid of Honour, 2026)

These are not diss tracks, not “wounded rapper rebuilding” tracks, not Toronto-anchoring tracks. They are R&B and club material, dominated by the May 2026 triple drop’s pop and PartyNextDoor-influenced sides (Habibti, Maid of Honour). The most “pre-sounding” tracks are from his rap and reflective canon — Headlines, Furthest Thing, Keep The Family Close, The Ride, Survival.

What the embedding shift actually captures is not “Drake responded to Kendrick” but “Drake’s post-event catalog is genre-skewed toward R&B and club material relative to his pre-event catalog.” Which is true — Habibti leans heavily into PNDR&B, Maid of Honour leans pop, and Iceman is the rap entry in the trio. Pre-event Drake spans his rap albums (Take Care, Nothing Was The Same, Views, Scorpion), his fragments (More Life, Care Package), and his earlier R&B detours — but the mix is more rap-heavy.

The cultural narrative says: Drake’s persona changed because of the beef. The first reading of the data says: Drake’s catalog composition changed in the post-event window. So the data, at this point in the analysis, cannot distinguish whether the composition change is a response to Kendrick or just normal artistic evolution that happened to coincide with the May 2024 cutoff.

That was the read I shipped with the first version of this post. A reader could reasonably ask: have you actually checked by stratifying on genre? The right move, before letting “genre confounder” stand as the final word, was to run the within-genre version of the test.

7. The within-genre re-test — at the song level, the genre confounder doesn’t explain it away§

I hand-tagged every album in the data with one of three labels — rap, rnb_pop, mixed — and re-ran the centroid permutation test inside each group. The full mapping lives in albums_genre.json; the borderline cases (Take Care, Nothing Was The Same, Views, More Life, Scorpion, Care Package, Certified Lover Boy) all went into mixed.

Genre breakdown of the 338 valid songs:

The mixed bucket happens to contain zero post-event tracks — none of Drake’s split-genre projects landed in the post-Kendrick window. The two real comparisons are within-rap and within-rnb_pop.

Two of these pass Bonferroni even when I count all four as a family ($\alpha = 0.05/4 = 0.0125$): within_rap at $p = 0.0034$ and rap_or_mixed at $p < 0.0001$. The third, “exclude the triple drop entirely,” is borderline ($p = 0.026$) but reassuring — even with the May 2026 release stripped out, the remaining 13 post-event tracks (the diss singles, 100 GIGS, $ome $exy $ongs 4 U, the 2025 singles) still sit measurably off from the pre-event centroid.

So at the song-level reading, the genre confounder is not the whole story. Drake’s rap material, on its own and balanced against itself, also appears to move. The §6 reading was too strong under that test. The honest revised song-level picture:

• The R&B-heavy May 2026 triple drop does inflate the unconditional embedding shift — that part of §6 was right.
• But the shift is not just the triple drop: with the R&B/pop sides removed entirely, Drake’s rap material still sits significantly off from his pre-Kendrick rap centroid under song-level permutation. Whether the same is true under a more honest inferential unit is the question §8 settles.

This looks, on song-level permutation, like a real semantic shift in his rap output — exploratory but defensible, and not detected by the four pre-registered surface lexicons. Whether the shift is caused by Kendrick or by other 2024–2026 personal and industry factors is unknowable from this data alone, and it is not reducible to “he just put out more R&B.” But before drawing that as the load-bearing conclusion of the post, one robustness check is owed. The pre-event “rap” bucket also inherits an event-window quirk worth naming: Push Ups (2024-04-19) and Family Matters (2024-05-03) are Drake’s own Kendrick diss tracks, but they fall before the May 4 cutoff and are tagged rap. They are the most defensive-deixis-dense, Toronto-anchoring, proper-noun-heavy tracks in his catalog, and they sit on the pre-event side. The pre-event rap centroid therefore already contains exactly the persona shape the post-event window was supposed to find. That biases the surface tests in §4 toward null in the predicted direction and the within-rap embedding test in §7 toward looking like a shift. The pre-registration locked the event date and I’m not changing it, but the bias is worth flagging.

More importantly, every embedding test in §5 and §7 permutes individual songs as if each were an iid observation. They are not. Songs cluster inside albums — same producer, engineer, recording session, thematic frame, often shared writers. The §8 robustness check moves permutation to the level of the release.

8. Five robustness checks that eat the headline§

The §5 / §7 song-level shift looked real because the test treated 338 songs as 338 independent observations. Below, five checks ask the question correctly. In order: (a) album-level cluster permutation — permute whole releases instead of songs; (b) a second encoder — re-run under all-mpnet-base-v2 to rule out a MiniLM-specific artifact; (c) ±6-month sliding cutoff — show the verdict isn’t pinned to one arbitrary date choice; (d) 23 placebo time-cuts — calibrate the May 2024 cluster p against what non-event dates in the same catalog produce; (e) peer-artist control — run the identical pipeline on J. Cole at the same cutoff and check whether the pipeline can detect anything at all in a catalog where no Kendrick-shaped event happened. Every check returns null. Cluster perm comes first because it’s the largest single methodological correction.

Album-level cluster permutation: the primary fix§

Every embedding test so far treats individual songs as exchangeable. They are not. Drake doesn’t independently sample 338 songs from a generative process; he produces albums (and a handful of one-off singles). Inside an album, songs share a producer roster, an engineer, a recording session, a thematic frame, often a co-writer pool. Embedding space is sensitive to all of that. Two songs from Iceman will sit closer to each other in 384-dim MiniLM space than two random Drake songs simply because they were made together, regardless of any Kendrick-effect.

Song-level permutation ignores this. The null distribution under random pre/post song relabeling is narrower than the true null under “what if a different set of releases had been the post-event ones,” which is the question the analysis is actually trying to answer. The standard fix in clustered data is to permute at the cluster level. In this dataset, no album straddles the 2024-05-04 boundary, so album-as-cluster labels are clean: 41 pre-event albums, 9 post-event albums.

I re-ran the four embedding tests with whole albums (not songs) randomly assigned to pre/post in the permutation null. Same 5000-perm Monte Carlo, same observed L2 distances, only the unit of label permutation changed:

The headline finding from §5 (p = 0.0002) collapses to p = 0.11. The §7 within-rap result (p = 0.0034) collapses to p = 0.35. Even the strongest variant, rap_or_mixed excluding all R&B, drops from p < 0.0001 to p = 0.069. None of the four embedding tests survives once permutation respects the obvious clustering structure.

Why such a dramatic swing? Because the post-event corpus consists of only 9 distinct release events. Within rap, only 6. The §5 song-level test was effectively asking “is this set of 55 songs different from 283 random songs?” — and answering yes, in large part because the 55 songs share their few albums’ embedding signatures. The album-level test asks the more meaningful question: “is this set of 9 release events different from 9 random release events drawn from the catalog?” The answer, at this sample size, is “we can’t tell.”

Two readings of this result, and I think honesty requires holding both:

One: the embedding shift is an artifact of song-level pseudo-replication. Drake’s rap output didn’t change after Kendrick in any way the MiniLM centroid can pick up at the appropriate inferential level. The §7 within-rap finding was a within-album-cluster echo, not a between-period effect.

Two: the embedding shift may well be real, but with only 9 post-event albums the test is severely under-powered at the cluster level. p = 0.11 unconditional is not “no effect” — it’s “this dataset has 9 post-event clusters and we cannot statistically distinguish 0.158 from drift at that sample size.” More post-event releases over the next 2-3 years would settle this.

The placebo calibration below picks between these two readings sharply. First a quick check that the null isn’t a property of the encoder.

Encoder robustness: same conclusion under MiniLM and mpnet§

A standard reviewer concern with any single-encoder finding is “your result is a property of the encoder, not the data.” all-MiniLM-L6-v2 is a 384-dim STS-trained model; a paper-grade version of this study would show the same test under at least one independent encoder. I re-ran the unconditional and within-rap cluster perm with all-mpnet-base-v2 (768-dim, different family) using the same chunking and L2-normalization.

Both encoders place the unconditional cluster p just outside α = 0.05 (0.10–0.11) and both put within-rap firmly non-significant (~0.35). The mpnet test produces a slightly smaller L2 distance in both cases, but the ordering and verdict do not change. The §8 conclusion is not a MiniLM artifact.

Sliding-cutoff sensitivity: the verdict isn’t tied to one date choice§

A skeptical reader could ask whether the null hinges on the precise May 4 cutoff — for example, “if you’d anchored at Kendrick’s euphoria drop on March 22, you might have seen a shift.” The pre-registration locked the date and there’s no degree of freedom to revisit, but I can ask how the cluster test behaves as the cutoff slides by ±6 months. None of these are valid alternative event dates; the exercise is purely a calibration of how brittle the verdict is to date choice.

The cluster p climbs from 0.058 (six months early) to 0.222 (six months late), but never crosses α = 0.05. The smallest p anywhere in the window is at the earlier cuts, where Drake’s own diss tracks (Push Ups, Family Matters) move into the post-event side and modestly inflate the centroid difference — yet even there the test does not clear the line. The L2 distance itself varies by less than 0.012 across the entire ±6-month window. No reasonable date choice in this neighborhood produces a significant cluster-level shift.

Placebo time-cuts: how unusual is the May 2024 result?§

A cluster p-value of 0.11 is meaningful only relative to what cluster p-values look like at non-event dates in the same catalog. If random 6-month-spaced cut dates also produce p ≈ 0.1 and L2 ≈ 0.16, the May 2024 result is unremarkable. If they produce much larger L2 and much smaller p, the May 2024 result is meaningfully less shifted than a typical cut — i.e., post-Kendrick Drake is closer to his pre-event self than the average 2-year window comparison would suggest.

I pre-committed (in placebo_cuts.py) to 23 candidate cut dates, every 6 months from 2012-07-01 through 2023-07-01, leaving a buffer of more than 9 months before the real event. For each candidate, the same album-level cluster permutation is run with the same embedding matrix; only the cut date moves.

The May 2024 cut produces a smaller L2 than 19 of 23 placebo cuts (83rd percentile from the small side), and a larger cluster p-value than all 23 placebo cuts (100%). Including placebos that mirror the real cut’s imbalance — 9-album minority side, 41-album majority side — those still produce L2 ≈ 0.22 (vs. 0.158 at May 2024), so the result is not simply a small-sample-noise artifact.

What this means for the two readings above: reading one (signal is genuinely absent) gets meaningful support; reading two (signal is real but under-powered) loses support. If the post-Kendrick effect were real but under-powered, you would expect its L2 to sit near or above typical placebo L2s, with the cluster p-value held back from significance only by the small post-event sample. Instead, the L2 itself is in the bottom quartile of the placebo distribution. The dataset is consistent with “no detectable Kendrick-specific shift” much more cleanly than with “real shift, low power.”

Control rapper: is the null Drake-specific?§

A reviewer’s next move on a single-artist null is: “you found nothing because your pipeline finds nothing.” The cleanest answer is to run the same pipeline on a peer artist not directly involved in the beef and check whether their May 2024 cut also produces a null. I fetched J. Cole, Lil Wayne, and Future the same way as Drake — Deezer enumeration plus lyrics.ovh, identical ≥ 80-token filter — and ran the identical album-level cluster permutation against the 2024-05-04 cut.

J. Cole is the only clean control of the three. He has more post-event songs than Drake (69 vs 55) across 5 post-event albums; his L2 distance is smaller than Drake’s and his cluster p of 0.76 is about as far from significance as a permutation test gets. The same pipeline applied to a closely-matched peer artist also fails to detect a May-2024 shift. The Drake null is not Drake-specific.

The other two artists are useful as a methodology stress-test more than as controls. Lil Wayne initially looked like a second null (cluster p = 0.87), but inspection revealed that his two “post-event” albums in this corpus are Tha Carter II (Deezer release date 2025-04-25, but lyrically the 2005 album: “32 I jump back like 33” on Fly In) and She Will (a 2011 single Deezer dates 2025-04-16). The Deezer endpoint exposes the storefront release date, not the original recording date — for catalog re-releases, Wayne’s “2025” lyrics are 2005 lyrics in a 2025 wrapper. The cluster permutation cannot tell the difference and treats those songs as post-event. His genuinely new 2024–2026 singles (Came Out A Beast, LIFESTYLE, Tha Carter VI material) mostly weren’t indexed by lyrics.ovh. The real lesson from Wayne is procedural, not substantive: any v2 of this pipeline must filter by original_release_date rather than the streaming-platform release date, and re-fetch from Genius for catalog coverage. As reported, Wayne’s row should be read as “data-quality null” not “negative control.” Future is the cleaner version of the same data-availability problem — only one post-event album indexed (MIXTAPE PLUTO, 17 songs), so the cluster permutation cannot run at all.

This generalization cuts two ways:

• In favor of the null reading: if both Drake and J. Cole — the only clean peer control — show no May-2024 shift, the most parsimonious story is that no rapper-of-this-tier saw a measurable lyric-content change at that date in this corpus. The “Kendrick changed Drake” narrative gets no specifically-Drake-shaped support.
• Against the null reading: if the pipeline produces a null on every artist of this size and release cadence, the test may just be structurally underpowered for any single artist at this sample size. The placebo time-cuts and sliding-cutoff sensitivity within Drake’s catalog already pointed in this direction; J. Cole extends it.

The honest synthesis is that at the appropriate inferential level, neither Drake nor a closely-matched peer shows a Kendrick-coincident lyrical shift, and the most one-evening NLP can say about the discourse claim is “the data does not corroborate it.” A second clean peer control would require either filtering Wayne and Future on original release date plus re-fetching their genuinely new 2024–2026 material from Genius, or picking peer artists whose Deezer catalogs aren’t dominated by recent re-releases. The next-most-leverage extension is exactly that re-fetch.

What this data does support cleanly:

• The four pre-registered surface predictions are confidently false.
• A song-level embedding test that ignores album clustering is statistically misspecified for this question and overstates the evidence by orders of magnitude.
• The May 2024 cut is calibrated against 23 placebo cuts and sits in the bottom of the L2 distribution — there is no positive evidence for a Kendrick-specific holistic shift beyond what random 2-year-window splits already produce.
• The same pipeline applied to J. Cole at the same cutoff produces a similar — actually firmer — null, ruling out “Drake-specific” as a back-door interpretation of the negative result.

Everything stronger than that is reaching.

9. Model-side robustness: held-out album train/test§

§8 was the inference-side fix — permute albums instead of songs. The model-side analog: train a classifier on a corpus that excludes specific albums, then ask whether it can recognize them as post-Kendrick when it sees them for the first time. If the post-event shift is a real generalizable signal, a model trained on the rest of the discography (including 8 of 9 post-event albums) should be able to label a 9th post-event album as post. If it cannot, the shift is album-specific.

The pre-committed protocol is leave-one-album-out (LOAO) over 15 albums: all 9 post-event releases plus 6 pre-event albums spanning Drake’s career as negative controls (For All The Dogs, Her Loss, Certified Lover Boy, Scorpion, Views, Take Care Deluxe). Two feature sets head-to-head: TF-IDF 1–2-grams and all-MiniLM-L6-v2 sentence embeddings. Both fitted with class-balanced logistic regression. Code in held_out_test_v2.py; artifact in out/held_out_loao.json.

Both feature sets converge at ~0.63 — above chance, but well below the song-level CV’s 0.76 (discriminative.py). The drop from 0.76 to 0.63 is the model-side mirror of p = 0.0002 → 0.11 in §8: the apparent signal shrinks materially the moment the held-out unit is the album, not the song.

The per-album posteriors are where it sharpens. MiniLM, post-event albums sorted by mean P(post-Kendrick):

The early-post one-offs classify as post-event; the May 2026 triple drop does not. All three of Iceman, Habibti, and Maid of Honour come back below 0.5, and Iceman — the largest of the three at 17 songs — is the most pre-Kendrick-looking album in the entire held-out set apart from Take Care and Views. TF-IDF agrees almost exactly: Iceman 0.391, Habibti 0.418, Maid of Honour 0.419 — same ordering, same verdict. Two feature sets, different inductive biases, same call.

The substantive read: whatever “post-Kendrick voice” the model finds is carried by a handful of short one-offs (Which One, DOG HOUSE, What Did I Miss?, 100 GIGS, $ome$exy $ongs 4 U) — most of them singletons. Drake’s biggest post-event project — the triple drop, 42 of 55 post-event songs by count — reads as pre-Kendrick to a classifier that has never seen it. This is the model-side restatement of §8 and the placebo result: the broader narrative-implied shift does not generalize across post-event releases. It localizes to short, mostly-singleton tracks that the model can latch onto only because they happen to be in the post-event window.

A more compact way to say it: there is no post-Kendrick Drake voice that a classifier can extract from the rest of his catalog and successfully apply to Iceman. The signal that exists in the data is small, scattered across one-off releases, and indistinguishable from what we’d expect from a model finding modest discriminative features in any small subset of a large discography.

This connects back to the §8 placebo result. If the classifier could pull “post-Kendrick voice” from the catalog and apply it to Iceman, the May 2024 cut would also pop in the placebo distribution. Neither happens. The two robustness checks — one on the inference side, one on the model side — agree on the same null.

10. What this stack of findings actually says§

Six honest results, in honest order:

1. The pre-registered surface-feature test is null. All four directional predictions about proper nouns, isolation pronouns, defensive deixis, and Toronto-anchoring fail to clear Bonferroni. The cultural narrative’s specific predictions about Drake’s surface vocabulary do not hold.

2. An exploratory song-level embedding test finds a shift. A 384-dimensional centroid moves well beyond what 5000 song-level permutation nulls produce ($p = 0.0002$). Suggestive but not pre-registered, and inferentially fragile.

3. The first interpretability pass blamed the May 2026 R&B-leaning triple drop. TF-IDF (AUC 0.76) and centroid-axis projection suggested a genre-composition artifact.

4. The within-genre re-test at song level also looks like a shift. Inside rap, song-level $p = 0.0034$. This is what convinced me the original “genre confounder” read was too strong.

5. Album-level cluster permutation withdraws every embedding p-value back above 0.05. Unconditional 0.11, within-rap 0.35, rap-or-mixed 0.07, exclude triple drop 0.07. The shift is sensitive to whether songs or albums are the unit of permutation, and the album-level answer is “we cannot reject the null.” A sliding-cutoff sensitivity around the event date confirms that no cut within ±6 months of May 2024 produces p < α = 0.05 either.

6. Placebo time-cut calibration sharpens the null. 23 candidate cut dates at 6-month intervals from 2012-07-01 to 2023-07-01. May 2024’s L2 = 0.158 is smaller than 19 of 23 placebos; its cluster p = 0.11 is larger than all 23. The signal at the real event date is less unusual than what random 2-year-window splits in Drake’s catalog already produce. This is positive evidence for the null, not just absence of evidence against it.

The one-paragraph synthesis I was tempted to write at the bottom of §7 — perception of change real, attributed mechanism false — does not survive the §8 robustness checks. The “perception of change is real” half of that dissociation was riding on a song-level p-value that doesn’t hold up under cluster inference and that the placebo calibration shows to be smaller than typical drift. So the cleaner claim is:

• The four most-cited folk mechanisms for “Drake changed after Kendrick” are falsified. Pre-registered, directional, all four null. This is the load-bearing result.
• The “holistic semantic shift” the exploratory analysis seemed to find does not survive cluster inference or placebo calibration. The May 2024 cut produces less pre/post divergence than nearly every random temporal split of Drake’s catalog. Whatever shape “Drake changed after Kendrick” takes in the discourse, it does not show up in a 384-dim sentence-embedding centroid distance at this sample size.

This is a sharper conclusion than the dissociation thesis would have allowed, and it’s what cluster inference plus placebo calibration actually shows. Two implications worth keeping.

First, on how listeners reason about artists. People are confident that Drake changed in specific surface-named ways — receipts, isolation, defensive deixis, Toronto-anchoring. None of those four shifted. And the broader semantic content the discourse implies should also have moved is, on a properly calibrated test, no more shifted than any random 2-year-window comparison of Drake to himself. The simplest reading is that the persona-shift narrative is a perceptual construction layered onto chart and cultural events, with the lyrics themselves carrying very little of it.

Second, on inference in low-frequency-event NLP studies. Song-level permutation is the default in lyric-analysis pipelines, and it produced a four-order-of-magnitude p-value that did not survive a one-line change in the permutation unit and was actively contradicted by placebo calibration. The methodological lesson is that with few post-event release events, cluster-level inference is not an optional robustness check — it is the appropriate primary test, paired with a placebo time-cut calibration to characterize what “unusual” actually looks like in the underlying catalog. A reasonable rule for studies of this shape: if your post-event corpus has fewer than ~20 distinct release events, report cluster-level inference as primary, and report a placebo distribution before claiming a directional effect.

This post’s headline is narrower than where I was an hour ago: the cultural narrative is confidently wrong about the four mechanisms it names, and the broader semantic shift it implies does not appear in this data on any defensible test.

11. What this study concludes§

Two findings stand at full strength after every robustness check:

1. The four pre-registered surface mechanisms are falsified. Proper-noun density, I/(I+we), defensive deixis, and Toronto-anchoring all return $p > \alpha_B$ in the directional ITS, with two of four observed in the opposite direction from prediction. The discourse-named mechanisms are not just unconfirmed — they are confidently rejected at α = 0.0125.
2. No embedding-detectable holistic shift survives appropriate inference. The song-level $p = 0.0002$ collapses to $p = 0.11$ at the album-cluster level, is robust to choice of encoder (MiniLM 0.11, mpnet 0.10), does not cross α = 0.05 at any ±6-month sliding cutoff, sits in the bottom quartile of L2 distances against 23 placebo cut dates, and is also not present in a closely-matched control artist (J. Cole, same pipeline, same cut, $p = 0.76$). The May 2024 cut is less pre/post-divergent than typical 6-month-spaced splits of Drake’s own catalog, and J. Cole’s L2 at the same date is smaller still.

What this rules out:

• Any surface mechanism in the discourse’s specific predicted directions.
• Any catalog-wide semantic shift at the level a 384–768-dim STS centroid is sensitive to, at the present sample size, that exceeds drift.

What this does not rule out:

• A real shift that a music- or hip-hop-specialized encoder would detect but STS encoders flatten (§12).
• A real shift restricted to Drake-attributed verses, which the current crowd-transcribed corpus cannot isolate (§12, §14).
• A real shift that emerges with ≥ 15–20 post-event release events instead of 9 (the structural ceiling on inference, §12).

A replication design that would convert this from “best-current-evidence null” to a confirmatory result is set out in §14. Two of those extensions are protocol changes a reviewer could ask for tonight (cluster-level pre-registration, audio-feature-derived genre tags); the rest are bounded by data availability.

12. Limitations, in descending severity§

• 9 post-event release events is structurally few, but the placebo calibration in §8 shows the signal isn’t there to detect. The album-level cluster perm alone gives p = 0.07–0.35 across all four embedding tests, which on its own could read as “underpowered” rather than “no effect.” The placebo time-cut calibration removes that ambiguity: at 23 candidate non-event cut dates the L2 distances are typically larger than 0.158 (median 0.171, max 0.220), even at placebos that mirror the real cut’s 9/41 album imbalance. So the limitation is no longer “we can’t tell because the sample is small” — it’s “even given the sample we have, the May 2024 cut does not stand out against drift.” A larger post-event window in 2027–2028 would still be the right paper-grade follow-up, but the current data is already informative.
• The May 2026 triple drop dominates the post-event corpus. 43 of 55 post-event tracks come from one day. The mix is unrepresentative of Drake’s long-run catalog by design — three simultaneously-released albums with very different genre targets. The §7 within-genre re-test addresses this partly, but the §8 cluster perm exposes the deeper issue: those 55 songs are really 9 release events, and 9 release events is not enough.
• Event-window quirk: the diss tracks are pre-event. Push Ups (2024-04-19) and Family Matters (2024-05-03) — Drake’s actual responses to Kendrick, and the most defensive/Toronto-anchored material in the catalog — fall on the pre-event side of the locked 2024-05-04 cutoff. This biases the four surface tests toward null in the predicted direction and slightly inflates the apparent within-rap shift in §7. The pre-registration locked the date, so this isn’t a fix; it’s a caveat. A v2 protocol should pick a cutoff that excludes Drake’s own beef contributions from the pre-event corpus.
• Monthly aggregation gives 6 post-event bins. Binding constraint on power for the surface tests. The pre-registration locked monthly bins; finer-grained aggregation would have violated protocol. Right answer is more time, not finer slicing.
• Hand-tagged genre labels are subjective. Take Care, Scorpion, Views, Certified Lover Boy, and Care Package all went into mixed and were thus excluded from the within-genre tests. A more rigorous version of §7 would use Spotify/Deezer genre tags or embedding-cluster labels. The current call is defensible but the sensitivity isn’t tested.
• Speaker attribution. “Drake-primary” tracks still contain guest verses. I strip the bracket headers ([Verse: Future]) but don’t filter to Drake-attributed lines, so features and Drake get mixed. This biases everything toward the average featured artist on each track — and given Drake’s collaborator pool may itself have shifted post-event (more PartyNextDoor, fewer Future-style features), “Drake’s lyrics changed” can collapse into “Drake’s guest roster changed.”
• Two encoders, both STS-trained. The cluster and placebo tests are reported under MiniLM and mpnet (§8) and agree on the null, but both are general-purpose STS encoders trained on similar data. A music- or hip-hop-specialized encoder (or a contrastively-trained one) could in principle separate lyrical features that STS models flatten. Whether such an encoder would reverse the null is unknown.
• Lyrics quality. Crowd-transcribed; Drake’s ad-libs, melodic delivery on Habibti, and slang neologisms are systematic transcription challenges. Spot-checks on a 10% sample looked clean enough.
• Genius HTML augment is a logged protocol deviation. Coverage only; no predictions or model changed. Stripping the source-asymmetric boilerplate was necessary and the §4 surface tests still pass through it unchanged.
• Lexicon-based metrics are brittle by design. A 2024 phrase Drake might use to deflect that isn’t in my regex list won’t count. An embedding-based pre-registration in a follow-up would be less brittle, but harder to interpret without an interpretability step like §6.

13. Reproducibility§

The full project lives at ~/hci/drake-nlp/. End-to-end, no API token required:

uv sync
uv run python -m spacy download en_core_web_sm
uv run python fetch_deezer.py        # ~7 min: Deezer + lyrics.ovh
uv run python genius_augment.py      # ~1 min: fills gaps from Genius HTML
uv run python clean_jsonl.py         # ~1 sec: strip cross-source boilerplate
uv run python analyze.py             # ~20 sec: spaCy POS+NER, regex, TTR
uv run python its.py                 # ~5 sec: pre-registered ITS
uv run python embed.py               # ~30 sec: MiniLM centroid + permutation
uv run python emotion.py             # ~30 sec: distilroberta emotion mix
uv run python vocab.py               # ~10 sec: TTR + MTLD
uv run python discriminative.py      # ~40 sec: TF-IDF logreg + axis projection
uv run python genre_controlled.py    # ~40 sec: within-genre centroid permutation
uv run python cluster_perm.py        # ~45 sec: album-level cluster permutation (§8)
uv run python multi_encoder.py       # ~3 min:  MiniLM + mpnet side-by-side (§8)
uv run python placebo_cuts.py        # ~3 min:  placebo time-cut calibration (§8)
uv run python cutoff_sensitivity.py  # ~1 min:  ±6-month sliding-cutoff sensitivity (§8)
uv run python fetch_artist.py --artist-id 339209 --name "J. Cole" --out data/jcole.jsonl
uv run python control_compare.py     # ~1 min:  peer-artist controls at the same cut (§8)

Pre-registered feature definitions, exactly as run:

I_TOKENS  = {"i","me","my","mine","myself","i'm","i've","i'd","i'll"}
WE_TOKENS = {"we","us","our","ours","ourselves","we're","we've","we'd","we'll"}

DEFENSIVE_PATTERNS = [
    r"\bthey say\b", r"\bthey said\b", r"\bthey think\b",
    r"\brumors?\b", r"\bgossip\b", r"\blies\b", r"\blyin'?\b about",
    r"\bcap\b", r"\bno cap\b",
    r"\bsnitches?\b", r"\bsnitch(?:ed|ing)?\b",
    r"\bhaters?\b", r"\bhatin'?\b",
    r"\bdon'?t believe\b",
    r"\bnever did\b", r"\bnever said\b",
]
TORONTO_PATTERNS = [
    r"\btoronto\b", r"\bovo\b",
    r"\bthe 6\b", r"\bthe six\b", r"\b6ix\b", r"\b416\b", r"\b647\b",
    r"\bcn tower\b",
    r"\bscarborough\b", r"\bnorth york\b", r"\betobicoke\b", r"\bmississauga\b",
    r"\bcanad(?:a|ian)\b",
]

ITS model:

# t: months since first observation
# D: 0 pre-event, 1 from event month onward
# P: 0 pre-event, (t - t_event) from event month onward
X = np.column_stack([np.ones_like(t), t, D, P])
model = sm.OLS(y, X).fit(cov_type="HAC", cov_kwds={"maxlags": 3})
# Reported: β₂ = params[2] (level shift), β₃ = params[3] (slope change)

Embedding test (song-level permutation, §5):

enc = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
X = np.vstack([enc.encode(chunks_of(lyrics), normalize_embeddings=True).mean(0)
               for lyrics in songs])
X = X / np.linalg.norm(X, axis=1, keepdims=True)

obs = np.linalg.norm(X[~post].mean(0) - X[post].mean(0))
null = [np.linalg.norm(X[~p].mean(0) - X[p].mean(0))
        for p in random_relabel(n_post, n_perm=5000)]
p_perm = (np.array(null) >= obs).mean()

Album-level cluster permutation (§8). Same obs, only the unit of the permutation null changes:

# album_idx[i] = integer id of the album that song i belongs to
unique_albums = np.unique(album_idx)
n_albums = len(unique_albums)
n_post_albums = int(album_is_post.sum())   # 9 of 50, unconditional

null = []
for _ in range(5000):
    perm_post_albums = rng.choice(n_albums, n_post_albums, replace=False)
    album_label = np.zeros(n_albums, dtype=bool)
    album_label[perm_post_albums] = True
    song_mask = album_label[album_idx]    # lift album label to songs
    null.append(np.linalg.norm(X[~song_mask].mean(0) - X[song_mask].mean(0)))
p_cluster = (np.array(null) >= obs).mean()

Pre-registration JSON (predictions, decision rule, stop rules, protocol-deviation log) lives at predictions.json. The single logged deviation is the Genius-HTML augment described in §3.

14. Open questions and replication design§

A v2 of this study, ranked by impact:

1. Cluster-level inference as the pre-registered primary, with placebo calibration alongside. Album-level permutation only. Song-level perm reported as a descriptive complement, not as a confirmatory statistic. Placebo time-cut pass moved into the pre-registered protocol rather than added as exploratory. Single highest-impact protocol change.
2. Wait two more years. With ~15–20 post-event releases (the realistic 2027–2028 count given Drake’s release cadence), the cluster-level test has a real chance of reaching power. With 9, it doesn’t matter what else you fix. This is the binding constraint.
3. More clean control rappers. §8 already includes a clean J. Cole control (cluster p = 0.76 at the same May 2024 cut — firmer null than Drake’s). Lil Wayne was attempted as a second control but his post-event corpus was dominated by Deezer re-releases of 2005 / 2011 material; Future had only one post-event album indexed. Extending this to Future, Lil Wayne, and Kendrick himself on Genius lyrics filtered by original release date would let the analysis distinguish three scenarios: (a) only Drake is affected, (b) all peer-tier rappers are affected, or (c) the test is structurally under-powered at this sample size for any artist. The current data is consistent with (b) or (c); a 5–6 artist comparison on properly-dated lyrics would separate them.
4. Speaker filtering. Use Genius’s verse-level annotations to score Drake-attributed lines only. Important because Drake’s collaborator pool may have shifted post-event in a way that confounds “his lyrics changed.” Attempted for this post — only ~15% of records in the cached corpus retain [Verse: …] markers (lyrics.ovh stripped them), so the analysis is deferred. A paper-grade run requires re-fetching the full Genius HTML and a robust verse-level parser.
5. Music- or hip-hop-specialized encoder. §8 already shows the null under MiniLM and mpnet, but both are STS-family encoders. A contrastive or domain-specialized encoder could pick up lyrical features that STS models flatten. The right baseline is a paper-grade contrastive encoder fine-tuned on rap-adjacent text — and ideally a head-to-head against the STS baseline so the reader can see what each architecture is sensitive to.
6. Pre-registered, audio-feature-derived genre tags before unblinding. Replaces the hand-labelling in albums_genre.json.
7. Event window that excludes Drake’s own beef contributions from “pre.” Push Ups and Family Matters belong on the response window’s side, not opposite it.

A separate question this pipeline can address with a binary pre-registrable answer: can a classifier recover Drake’s own 3-way partition (Iceman / Habibti / Maid of Honour) from verse text alone? §6’s TF-IDF evidence suggests it can. Different study, different post.

Bottom line: an unsupervised NLP pass over Drake’s whole discography — surface metrics, sentence embeddings, emotion classification, two encoders, album-level cluster permutation, 23 placebo time-cuts, a J. Cole control, and a leave-one-album-out classifier test — looks for the changes the discourse insists happened after Kendrick. Nothing in the data corroborates them. Receipts, isolation, defensive deixis, Toronto-anchoring all return null on directional tests. The one apparent embedding shift collapses under the appropriate cluster-level inference, is less unusual than typical placebo cut dates, and is not even Drake-specific — J. Cole’s L2 at the same cut is smaller (0.110 vs 0.158) and his cluster p is 0.76. And on the model side: a held-out classifier trained on Drake’s catalog without the May 2026 triple drop cannot recognize Iceman, Habibti, or Maid of Honour as post-Kendrick (pooled LOAO AUC ~ 0.63 vs 0.76 song-level; all three triple-drop albums score below 0.5). The takeaway isn’t really about Drake — it’s that with this little post-event data, song-level permutation can manufacture orders of magnitude of apparent evidence that cluster-level inference, placebo calibration, peer-artist control, and held-out classifier tests all refuse to confirm.