Systematic partisan content material skews in TikTok through the 2024 US elections

Ethics assertion

All through this research, we took care to observe related moral requirements. Using sock puppet accounts is a longtime analysis method for investigating personalization and bias on web platforms^12,24,45, supplied it’s strictly for noncommercial, public-interest functions and doesn’t compromise consumer privateness. Earlier work and authorized precedent have indicated that the Phrases of Service of social media platforms, which can prohibit automated entry to content material, don’t essentially battle with gathering publicly out there knowledge for analysis goals^46,47. Given our give attention to potential impacts on democratic processes and political discourse, this venture falls below well-recognized tutorial exceptions for finding out on-line data ecosystems. Furthermore, we minimized the danger of privateness breaches or business hurt by limiting our knowledge assortment to publicly out there content material. Though the experiment itself didn’t embrace any human members, we obtained knowledgeable consent from the human annotators for our LLM classification validation duties.

For the survey element of this research, the survey was deemed exempt by the authors’ institutional evaluate board (IRB protocol no. HRPP-2025-69).

Experimental setup

On this experiment, we aimed to measure the speed at which movies of a sure political leaning seem within the suggestions of TikTok, within the context of US politics. To take action, we use bots that simulate TikTok customers by watching each predefined sequences of movies of a given political leaning (the ‘conditioning’ stage), after which subsequently watching advisable movies on ‘For You’ web page (the ‘advice’ stage) of a given bot. Every experimental run, comprising these two phases, lasts for a 1-week interval.

Over the period of 27 weeks, 567 experiments had been performed. Particularly, every week, 21 new TikTok accounts are created by randomly combining the most typical American first and final names from ref. ⁴⁸ and assigning an age between 22 years and 24 years. This allowed every account to impersonate a possible voter for the US presidential election more likely to be lively on TikTok. This age vary was additionally an intentional choice made to standardize the age of the consumer throughout experimental situations. Furthermore, our choice to pick this age vary was guided by knowledge suggesting that the 18–24 years age bracket had the most important share of customers on TikTok within the USA as of 2022 (35%), though the older 25–34 years age bracket now occupies the most important share as of late 2024 (ref. ⁴⁹). The 21 accounts created every week are break up into one among 9 experimental situations, that are outlined by two attributes. The primary attribute is the state to which the bot is manually geo-located, which is both New York, Texas or Georgia (Georgia was largely considered a key swing state for 2024 US presidential elections, with a slender 0.23% margin in favour of Joe Biden within the earlier 2020 elections). We selected New York, Texas and Georgia as a result of, on the idea of the 2020 presidential outcomes, they function clear prototypes of a reliably Democratic state, a reliably Republican state and a aggressive swing state, respectively, within the 2024 electoral panorama. Sensible constraints additionally formed this selection, as solely these three places had been concurrently out there and steady throughout our VPN infrastructure. Among the many possible choices, they maximized geographic and partisan range, however we warning that our findings shouldn’t be presumed to generalize past these states. The second variable is the political leaning of the movies the bot watches within the conditioning stage. The movies watched within the conditioning stage are printed by both recognized Democrat-supporting channels or Republican-supporting channels. Lastly, every week, three bots geo-located to Georgia bypass the conditioning stage of the experiment and transfer on to the advice stage. That is finished to gather suggestions made to customers who would not have a specific curiosity in politics. A abstract of the experimental situations is supplied in Supplementary Desk 2.

Supplementary Fig. 4 reveals a extra detailed timeline of a bot throughout a given experimental run. Though our unique design contemplated a completely crossed 3 (State: New York, Texas, Georgia) × 3 (Partisan seed: Democrat, Republican, Impartial) construction, we finally applied a diminished model with seven situations. Specifically, neutral-seeded accounts had been deployed solely in Georgia. This choice was pushed by sensible constraints: increasing to 9 concurrent experimental situations required extra units to run concurrently, which elevated the danger of detection and deactivation by the platform safeguards of TikTok. We prioritized the inclusion of a impartial baseline in Georgia, a prototypical swing state, given its strategic worth for deciphering partisan results in a politically heterogeneous context. Though this design doesn’t permit for full within-state comparisons between impartial and partisan accounts in New York and Texas, we observe this as a limitation and interpret state–partisanship interactions with applicable warning.

Pre- and post-experiment protocols

TikTok infers consumer location via the GPS or community (IP) geo-location of the gadget⁵⁰. This required us to dedicate an Android smartphone, specifically, Samsung Galaxy A34 5G, to every of the 21 accounts created each week. Earlier than every experiment, we managed gadget geo-location throughout three goal states utilizing a mixed method of GPS mocking and VPN tunnelling. Particularly, we used AnyTo⁵¹ for GPS coordinate spoofing, setting New York bots to ⟨40.7308, −73.9976⟩ in Manhattan, New York Metropolis; Texas bots to ⟨33.148, −96.638⟩ in Collin County; and Georgia bots to ⟨33.961, −84.537⟩ in Cobb County. These particular places had been chosen as counties that voted strongly Democrat, Republican or had been a detailed name within the 2020 US presidential elections, respectively. Moreover, to align the community id of every bot with the geo-location of the supposed state, we tunnelled the general public IP handle of every telephone to one among three customized VPN servers we hosted on third-party cloud suppliers. We averted business VPN providers to attenuate the danger of TikTok figuring out the IPs as digital. We put in TikTok from Google Play Retailer solely after the GPS and IP handle of every telephone had been appropriately modified. On the conclusion of a weekly experiment, we factory-reset each telephone earlier than starting the following spherical of the experiment. This step ensures that any TikTok-related cache is cleared and doesn’t affect the following experiments performed on the identical telephones. Lastly, all telephones operated on Android 13, which re-randomizes the MAC (medium entry management) handle each 24 h (ref. ⁵²), precluding the potential of device-level monitoring or bot detection of TikTok all through a weekly experiment.

Conditioning stage

Within the conditioning stage of the experiment, bots watch a sequence of movies printed by TikTok channels aligned with both Democratic or Republican political leanings. To gather the channels that the bots would watch on this stage, channels had been compiled iteratively by trying to find politically charged key phrases, corresponding to ‘Trump’, ‘Biden’ or ‘Kamala’ on the search bar of TikTok, other than the phrases ‘Democrat’ and ‘Republican’. From there, the channels printed movies supporting every candidate had been compiled and verified by the authors to suit the next standards: (1) a lot of the movies printed by the channel concern political content material; and (2) all of the political movies printed by the channel are aligned with both the Democratic or Republican political events.

In complete, 54 candidate channels had been compiled throughout each political social gathering affiliations. Every channel related to a given political social gathering was then matched with a channel from the alternative political social gathering based mostly on its variety of followers and cumulative variety of likes utilizing Euclidean distance matching. This was finished to account for the chance that the power of the political conditioning of a given bot could also be partially attributed to the recognition of the channels it watches within the conditioning stage. After matching, the ten pairs of channels with the smallest Euclidean distance had been chosen. Supplementary Desk 36 particulars the channels used within the conditioning stage, exhibiting 12 pairs in complete, as two channels (donaldtrumpwasright and kayetriots) grew to become not out there (deleted or made non-public) throughout our experiments. As such, these two pairs had been changed with the brand new pairs that had the smallest Euclidean distance. The accounts of the principle political candidates (kamalaharris and realdonaldtrump) didn’t exist when the experiment started in Might, and therefore, weren’t included as conditioning channels. Kamala Harris formally joined TikTok in July, whereas Donald Trump joined the platform in June.

In every experiment, a bot is conditioned to lean both Democratic or Republican by watching as much as 50 most up-to-date movies from eight randomly chosen channels aligned with the respective political social gathering. The goal of every bot watching 400 conditioning movies in complete was not at all times achieved, as proven in Supplementary Desk 37. On common, Republican bots considered fewer conditioning movies due to a number of of the Republican accounts having fewer than 50 complete printed movies, notably through the first few months of the experiment. To account for this, the variety of movies watched through the conditioning stage was managed for throughout all analyses that in contrast advice charges throughout experimental situations. Each video through the conditioning stage was watched for 1 min, making certain constant publicity to the content material from the chosen channels. After finishing the predefined set of movies, the bot would then ‘sleep’ for twenty-four h, throughout which it could not open or work together with TikTok. This pause was applied to simulate sensible human viewing patterns, minimizing the danger of bot detection that would consequence from consuming an extreme variety of movies in speedy succession.

Advice stage

Following the conditioning stage, every bot within the experiment transitions to the advice stage, during which they watch movies that seem on their For You web page. The For You web page is the default interface on TikTok, during which customers can watch movies advisable to them based mostly on their pursuits as implicitly decided by the algorithm of TikTok. On this stage, every video is watched for as much as 10 s, after which the URL of the video is retrieved. To bypass the bot detection mechanism of TikTok, solely the primary 10 advisable movies had been watched per hour, adopted by a 60-min sleep. Furthermore, the TikTok app was reloaded earlier than each hourly session to keep away from watching movies preloaded through the earlier iteration.

All URLs collected in a given experimental run are then used to retrieve the metadata of the video, together with the writer of the video, the outline of the video and its embedded transcript, if out there. Of the 176,252 distinctive movies watched, 40,264 had a transcript out there, and it’s this set of 40,264 movies that we analyse within the the rest of this research. Within the ‘Information Representativeness’ part, we present that this pattern is consultant of your entire dataset of advisable movies.

Experimental run validation

Naturally, with audit experiments corresponding to this one, experimental failures are inevitable due to points such TikTok classifying the account as a bot and subsequently suspending the account, or web outages. As such, to make sure the identical quantity of advice publicity for Democrat- and Republican-conditioned bots, we match bots of reverse conditioning in every state throughout a given experiment week. Consequently, for every pair of bots, we solely think about the primary n suggestions made to every bot, the place n is the lesser of the entire variety of suggestions made to both bot. Moreover, we solely think about pairs of bots that watched not less than 150 movies every. This threshold is a deviation from our preregistration, which set the brink at 500, due to our preliminary overestimation of the variety of movies bots would watch throughout a given experimental run.

There have been a handful of weeks during which the bots failed to fulfill our inclusion threshold. First, on the finish of July and the start of August, bots with a Texas geo-location misplaced web connectivity whereas the authors weren’t out there to are inclined to the bots. As such, we had been unable to gather knowledge from Texas throughout this time. In one other week, some accounts had been acknowledged as bots by the bot detection algorithm of TikTok and subsequently suspended earlier than reaching the brink of 150 movies. Of the 567 experiments performed, 323 met our inclusion standards. Supplementary Desk 3 lists the entire variety of bots that met the inclusion standards on a weekly foundation throughout the totally different bot conditioning and geo-location courses.

Ideological stance classification

To analyse the ideological stances current within the video content material, we applied a three-step classification method utilizing an ensemble of LLMs comprising GPT-4o, Gemini Professional and GPT-4. First, every video transcript was evaluated for political content material utilizing the immediate: ‘Given the next video transcript, do you suppose the subject is political?’

For transcripts recognized as political, we performed two further classification steps. The primary assessed election relevance via the immediate ‘Given the next video transcript, do you suppose the subject is said to the 2024 US election or associated to Donald Trump, Kamala Harris, Joe Biden, JD Vance, or Tim Walz? Reply with solely Sure or No’.

The second step decided the partisan stance utilizing the immediate ‘Given the next video transcript, classify the transcript into one of many following classes:

Anti Democrat

Anti Republican

Professional Democrat

Professional Republican

Impartial’.

The distribution of classifications throughout these classes is introduced in Supplementary Desk 11.

All through our research, we frequently group the classes ‘Anti-Democrat’ and ‘Professional-Republican’ below the broad class of ‘Republican-aligned’, and equally group the classes ‘Anti-Republican’ and ‘Professional-Democrat’ below the class of ‘Democrat-aligned’.

To make sure classification reliability, we used a consensus-based method during which GPT-4 served because the tiebreaker in circumstances of disagreement between GPT-4o and Gemini Professional. The inter-model settlement charges for every classification job are detailed in Supplementary Desk 7. This ensemble methodology was chosen to mitigate particular person mannequin biases and improve the robustness of our ideological stance classifications.

Lastly, in spite of everything movies had been labeled, we excluded the ten commercial movies encountered by the bots throughout your entire period of the experiment, which had been labelled as political. Of those, just one had a non-neutral ideological stance (‘Anti-Democrat’). Moreover, not one of the 10 ads was printed by an official political candidate channel.

Ideological stance validation

To validate our automated classification method, we performed a human annotation research on a subset of 500 randomly chosen transcripts. Three impartial annotators, all political science undergraduate college students, had been recruited to carry out the classification duties. The annotators had been permitted to make use of internet searches to analysis unfamiliar subjects or references inside the transcripts, making certain knowledgeable labelling selections. Every transcript acquired impartial classifications from all three annotators following the identical categorical framework used within the LLM classification. The outcomes of this validation research, together with inter-rater reliability metrics, human majority–LLM majority classification accuracy, Cohen’s κ coefficient evaluating human-majority and LLM-majority selections, and the F1 scores of the LLM ensemble, are introduced in Supplementary Desk 8, whereas Supplementary Desk 9 particulars these scores for every mannequin individually.

Channel classification

To establish channels doubtlessly leaning in direction of both the Democratic or Republican ideologies, we centered on channels that had printed movies beforehand labelled with a selected non-neutral ideological stance (for instance, ‘Anti-Democrat’, ‘Professional-Republican’, ‘Anti-Republican’ or ‘Professional-Democrat’). From this course of, we recognized 170 distinctive channels for evaluation. For every channel, we calculated the proportion of movies aligned with the ideology of a specific social gathering as a fraction of the entire movies collected for that channel. To make sure robustness in our classification, channels had been labelled as Democrat-aligned or Republican-aligned if greater than 75% of their analysed movies had been ideologically in step with one social gathering. For 85 channels with fewer than 10 distinctive movies collected through the experiment section, we used TikAPI (https://tikapi.io/) to retrieve as much as 30 further movies printed earlier than the election. These further movies had been processed via the identical transcript classification pipeline to find out their ideological stance. This step was essential to keep away from potential misclassification brought on by the distinctive movies watched by the bots of a restricted variety of channels. Supplementary Desk 38 presents a abstract of the labeled channels, together with the entire variety of channels in every class (Republican-aligned, Democrat-aligned and Impartial) in addition to the common proportion of partisan content material for every group. Lastly, we validated our 75% threshold by inspecting 13 channels falling between 60% and 75% alignment, which predominantly comprised conventional information shops (Day by day Mail, USA TODAY and Channel 4), discuss reveals (The Day by day Present, Don Lemon and The Downside With Jon Stewart), and journalistic content material that usually covers political subjects whereas sustaining some editorial steadiness. This distribution confirms that the 75% threshold successfully distinguished persistently partisan channels from these with occasional political lean (see Supplementary Desk 5 for an inventory of channels that didn’t meet the 75% threshold).

Remark classification and uneven homophily

To establish the ideological stance of the feedback on a given TikTok video, we observe an identical methodology to that of classifying a video. Particularly, we cross each the transcript of the video and the remark in query in a immediate to GPT-4o, which will be seen under: ‘Given the next TikTok video transcript and remark, classify the remark into one of many following classes.

Transcript of the video: TRANSCRIPT

Remark: COMMENT

Classes:

Anti Democrat

Anti Republican

Professional Democrat

Professional Republican

Impartial’.

The outputs of the mannequin had been validated by three impartial annotators, once more, all political science undergraduate college students. The outcomes of this validation course of are proven in Supplementary Desk 10.

To look at whether or not cross-party engagement patterns (‘uneven homophily’) might clarify the noticed skew seen in Fig. 2nd–f, we analyse the partisan composition of feedback on political movies. The important thing speculation is that, if Democrats systematically interact extra with Republican-aligned movies than Republicans interact with Democratic-aligned movies, Republican-aligned movies that entice extra Democratic-engaged feedback ought to in flip be advisable extra typically to Democratic bots.

Particularly, we gather a random pattern of 500 feedback from all political movies examined in our research and classify the content material of every remark within the context of that video. From this, we compute the proportion of feedback ideologically aligned with the video. We discover significant variations between Republican-aligned and Democratic-aligned movies within the composition of feedback, however not within the anticipated route. We discover that Democratic-aligned movies have a nominally larger proportion of Republican-aligned feedback than vice versa, though this distinction shouldn’t be statistically important (12.9% in contrast with 8.5%; two-sided χ² = 2.32, P = 0.13). Democratic-aligned movies even have a nominally larger proportion of co-partisan feedback than Republican-aligned movies, once more with out reaching statistical significance (63.1% in contrast with 51.6%; two-sided χ² = 1.97, P = 0.16). Nonetheless, Republican-aligned movies have a considerably larger proportion of ‘impartial’ feedback that aren’t explicitly partisan (44.6% in contrast with 26.3%; two-sided χ² = 7.53, P = 0.006). We additionally embrace this metric in our weighted sampling robustness verify (Fig. 2g), discovering comparable outcomes to our different noticed engagement metrics.

Conditioning stage validation

We take a number of steps to validate that the seeding delivered to bots of reverse social gathering alignments was equal. A abstract of this evaluation is introduced in Prolonged Information Fig. 1. First, we consider the proportion of movies immediately associated to the US election or main political figures (as decided by the second query of the transcript classification pipeline described above). As proven in Prolonged Information Fig. 1a, we discover that Democratic-aligned and Republican-aligned bots had been uncovered to a statistically comparable proportion of election-related movies (two-sided impartial t-tests; t = −0.478, P = 1.0).

Concerning optimistic and unfavourable partisanship, though we discover no statistically important variations between the 2 bot teams (two-sided impartial t-tests; optimistic partisanship: t = 2.19, P = 0.129; unfavourable partisanship: t = −1.7, P = 0.373; and impartial: t = 1.63, P = 0.426), we do nonetheless see a modest asymmetry, during which the Republican seed set contained extra anti-Democratic than pro-Republican content material, whereas the Democratic seed set was comparatively balanced. Though this distinction was not statistically important, it’s nonetheless vital to acknowledge. This asymmetry doesn’t replicate an intentional experimental design selection. Somewhat, it displays the availability of content material from the accounts chosen for seeding, during which anti-Democratic movies had been marginally extra prevalent among the many chosen Republican-aligned creators than anti-Republican movies had been among the many chosen Democratic-aligned creators.

To evaluate whether or not this distinction had downstream results on the proportion of unfavourable partisanship movies seen through the advice stage, we mannequin the probability of a advisable video being of unfavourable partisanship as a perform of the proportion of those movies seen through the conditioning stage of a bot. We discover that it’s neither the conditioning of the bot nor the proportion of unfavourable partisanship movies seen within the conditioning stage that considerably predicts a advisable video being of unfavourable partisanship. Somewhat, it’s the traits of the writer of the video (their follower depend and complete share depend), in addition to the normalized variety of feedback the video receives which can be important, suggesting that the hole in unfavourable partisanship suggestions is because of supply-side elements slightly than elements in our management within the conditioning stage (Supplementary Fig. 10). Nonetheless, we acknowledge that this imbalance might contribute, not less than partially, to the uneven outcomes we observe, and we spotlight this as a limitation of our design.

Second, we classify a random pattern of as much as 500 feedback posted below every of the partisan movies seen through the conditioning stage, utilizing the remark classification process described within the earlier part. As proven in Prolonged Information Fig. 1b, movies proven to Democratic- and Republican-aligned bots acquired comparable charges of each party-aligned and opposite-party-aligned feedback (party-aligned feedback: 60.2% in contrast with 59.9%, two-sided χ² = 0.008, P = 0.925; reverse party-aligned feedback: 8.9% in contrast with 6.8%, two-sided χ² = 0.18, P = 0.67).

Third, we compute the embedding of the transcripts of all movies inside our dataset utilizing the embedding mannequin proposed in ref. ⁵³, chosen as an open-source embedding mannequin with efficiency on par with state-of-the-art fashions. We then compute the centroids of the units of movies seen by Democratic-aligned and Republican-aligned bots through the conditioning stage of their respective audits. The projection of those cluster centroids into two dimensions utilizing principal element evaluation (PCA) is proven in Prolonged Information Fig. 1c. To higher perceive the extent to which the units of movies diverge semantically throughout partisan situations, we compute the axis of maximal variation between Democratic and Republican movies by taking the unit vector connecting their respective centroids within the unique embedding area. We then venture the corresponding centroids onto this axis, permitting us to quantify and visualize partisan separation in semantic area. This method allows us to evaluate whether or not the movies proven to Democratic- and Republican-aligned accounts are meaningfully distinct in content material, or whether or not they largely occupy overlapping areas of the embedding area. As proven in Prolonged Information Fig. 1d, the cluster centroids of conditioning-stage movies seen by Democratic-aligned and Republican-aligned bots had been roughly equidistant from the centroid of impartial movies, each when it comes to their projected positions alongside the axis of maximal variation (0.05 in contrast with 0.032), in addition to the common Euclidean distance between the person movies belonging to that cluster and the impartial centroid within the unique high-dimensional embedding area (0.446 in contrast with 0.443, t = 1.962, P = 0.05).

Taken collectively, these analyses counsel that the seeding movies proven to Democratic- and Republican-aligned bots had been largely equal in each content material and semantic distribution. Throughout a number of validation checks, together with election relevance, partisanship framing, viewers partisanship within the type of feedback and transcript-level semantic similarity, we discover little to no constant proof of systematic variations between the 2 teams. That is other than the pre-seeding matching steps taken to make sure that movies had been chosen from channels of an identical total reputation.

Cross-partisan advice regression mannequin

We additional formalize the robustness analyses utilizing a linear chance mannequin (LPM) on the bot-video advice degree. The dependent variable Y_ib is an indicator that video i advisable to bot b is a cross-partisan advice, outlined as a advice during which the partisan alignment of the video doesn’t match the conditioned partisanship of the bot. The important thing impartial variable is an indicator for whether or not bot b was conditioned on Democratic in contrast with Republican content material.

The baseline specification is

the place DemBot_b equals 1 for Democratic-conditioned bots and 0 for Republican-conditioned bots, X_ib is a vector of covariates, α is the intercept, β is the coefficient on the Democratic conditioning indicator (the parameter of major curiosity), γ′ is a vector of coefficients on the covariates X_ib, δ_b are bot fastened results, λ_w are week fastened results and ε_ib is the error time period. The covariate vector X_ib contains the variety of movies watched by the bot throughout its experimental run, the proportion of advisable movies with transcripts, and a set of video- and channel-level engagement measures for each conditioning-stage and recommendation-stage movies (for instance, likes, feedback, performs, share depend, follower depend and verification standing). We estimate the mannequin utilizing OLS with two-way cluster-robust commonplace errors by bot and week to account for within-bot and within-week dependence in suggestions.

As a result of many engagement covariates are extremely correlated, we assess multicollinearity utilizing variance inflation elements (VIFs). Impartial variables with VIF values higher than 5 are mixed right into a lower-dimensional index utilizing PCA; specifically, we summarize conditioning-stage video- and channel-level engagement variables right into a single mixed engagement rating used as a management within the regression. Supplementary Tables 20 and 21 report the VIF scores for all covariates.

For ease of interpretation, the main-text regression figures report outcomes from LPMs, whose coefficients will be learn immediately as percentage-point modifications within the final result (Supplementary Tables 22, 24 and 26). To evaluate robustness to functional-form assumptions, we additionally estimate the identical specs utilizing logistic regression and report these leads to Supplementary Tables 23, 25 and 27. Throughout all fashions, the signal and substantive significance of the important thing coefficients are largely comparable within the logit and LPM specs.

Given the massive variety of coefficients examined throughout associated fashions, we alter for a number of comparisons utilizing Benjamini–Hochberg (BH) corrections, which management the false discovery charge. We report each unadjusted and BH-adjusted P values in Supplementary Tables 22–27, and use BH-adjusted P values as the idea for statistical significance markers in Fig. 3e and Supplementary Fig. 1b.

Sensitivity evaluation

Though we now have proven proof that observable engagement metrics and viewership patterns can not clarify the ideological skew in Republican-aligned content material on TikTok, we can not rule out the chance that there are unobserved or latent engagement metrics or viewership patterns, recognized to the TikTok algorithm however unknown to us, that may clarify the hole we observe. As an example, a key engagement metric corresponding to ‘dwell time’, or how lengthy a consumer spends watching a specific video, shouldn’t be a publicly out there metric printed by TikTok. As such, we conduct a sensitivity evaluation to calculate how robust such an unobserved issue must be to completely clarify the obvious ideological skew of TikTok.

In Supplementary Desk 39, we present within the first column the true distinction between the Republican and Democratic movies with regard to their normalized values for a given engagement metric. Within the remaining 4 columns, we present the ratio between the distinction wanted to match the noticed skew and that of the true distinction in imply between Republican and Democratic movies in a given engagement metric. As will be seen, for a lot of the engagement metrics, it’s Democratic movies that are inclined to have larger engagement on common, with the exceptions being within the video remark depend, corresponding to depend, play depend, share depend and composite rating. Though the mix of those metrics produces the most important normalized Republican skew in engagement hole, this hole would should be 9.4 occasions as massive to completely clarify the noticed skew in advice charges seen in our experiments.

Information representativeness

Provided that our major evaluation all through this research focuses on movies with transcripts, we should be sure that this pattern is consultant of movies all through your entire dataset. To take action, we annotate a random pattern of movies advisable to every of the units of Republicans and Democrats through the experiment. Particularly, for every of those units, we analyse a random pattern of two,000 movies that don’t embrace transcripts to measure the next options of a given video: (1) whether or not the video is political in nature; (2) whether or not the video is anxious with the 2024 US elections or main US political figures; and (3) the ideological stance of the video in query (Professional Democrat, Anti Democrat, Professional Republican, Anti Republican or Impartial). This annotation job was once more accomplished by three political science undergraduates, with excessive settlement between annotators (Krippendorf’s α; Q1: α = 0.858, Q2: α = 1.0, Q3: α = 0.989). We discover that the inter-rater settlement when viewing the movies immediately, slightly than counting on the transcript alone (the outcomes of that are in Supplementary Desk 7), yielded larger settlement between the raters.

This course of permits us to check the distribution of political content material in movies with transcripts towards these with out transcripts. The outcomes of this comparability are proven in Prolonged Information Fig. 2. As will be seen, movies with no transcripts contained considerably fewer political movies on common relative to people who did embrace transcripts. This result’s maybe unsurprising on condition that movies that don’t embrace a transcript are additionally much less more likely to embrace dialogue (for instance, movies of landscapes or pets). Nonetheless, of people who had been political, we don’t discover important variations (computed via χ² checks) between transcript and non-transcript movies with regard to the proportion of political movies concerning the election, nor their distribution of Democrat-aligned, Republican-aligned or Impartial movies.

Subject evaluation

To establish the subjects mentioned in every political video, we use the methodology utilized in ref. ⁵⁴. Particularly, the authors used GPT-4 to categorise a video in one among a set of subjects, and totally validated their method, discovering accuracies higher than 0.9 for all subjects.

We barely modify the immediate utilized in ref. ⁵⁴ to permit the LLM to pick multiple matter choice. The immediate used will be seen under:

You might be an AI assistant skilled to take a look at social media posts and decide what the put up is about.

You’ll obtain the transcript of the video in query.

You’ll be given an inventory of subjects. Please inform us what this put up is about.

Transcript: [TRANSCRIPT]

Another issues to bear in mind:

• There may be an election in November, so many posts might be about that. The Democratic candidates had been Joe Biden and Kamala Harris, however Joe Biden dropped out and it’s Kamala Harris and Tim Walz. The Republican candidates are Donald Trump and JD Vance. The impartial candidates are Robert F Kennedy Jr (RFK Jr), Cornel West, and Jill Stein.

• If there are a number of subjects, decide all of the related subjects.

• If a put up appears political, first see if it goes into one other class. For instance a put up about politics and race would go into the race class. Fall again on politics if none different matches, so long as there’s something political.

• Except for immigration, most posts that reference a international nation will go into the final class.

Classes:

1. 1.

   Crime

   1. 1.1.

      Crime usually

2. 2.

   Surroundings

   1. 2.1.

      Local weather change

   2. 2.2.

      Different environmental points

3. 3.

   Immigration

   1. 3.1.

      Immigration usually

4. 4.

   Social points

   1. 4.1.

      Abortion and reproductive well being

   2. 4.2.

      Weapons and gun management

   3. 4.3.

      LGBTQ+ points, together with transgender points

   4. 4.4.

      Racial points, together with affirmative motion and racial discrimination

   5. 4.5.

      Schooling

   6. 4.6.

      Different social points, together with tradition warfare points, labor, and different social points that aren’t coated above

5. 5.

   Public well being

   1. 5.1.

      Covid, together with covid vaccines

   2. 5.2.

      Different vaccines

   3. 5.3.

      Different public well being points

6. 6.

   Economic system

   1. 6.1.

      Economic system usually

7. 7.

   Know-how

   1. 7.1.

      AI, LLMs

   2. 7.2.

      Crypto

   3. 7.3.

      Different know-how points

8. 8.

   Authorities, politics and elections

   1. 8.1.

      Assassination try on Donald Trump

   2. 8.2.

      Republican Nationwide Conference (RNC)

   3. 8.3.

      Democratic Nationwide Conference (DNC)

   4. 8.4.

      Biden dropping out of the presidential race

   5. 8.5.

      Different political or authorities associated posts that don’t match into different classes

9. 9.

   Worldwide points

   1. 9.1.

      Israel, Gaza or Palestine, together with something about Netanyahu or Hamas

   2. 9.2.

      Ukraine warfare

   3. 9.3.

      Something exterior the US or contain US international relations aside from Israel, Gaza, Ukraine, or immigration

10. 10.

    No matter

    1. 10.1.

       Not one of the above subjects

Utilizing the above immediate, the subjects mentioned within the 7,767 ideologically stanced TikTok movies in our dataset had been recognized. Supplementary Desk 40 describes the variety of movies falling below every class, break up by the ideological stance of the video.

Video advice counterfactual fashions

To confirm that the Republican skew noticed in our experiments shouldn’t be merely on account of variations within the engagement metrics of Republican- and Democratic-aligned movies or channels, we construct a collection of counterfactual fashions to foretell anticipated video suggestions based mostly on these engagement metrics. Particularly, we goal to compute the ideological content material that may come up if suggestions had been pushed purely by observable engagement, after which examine these anticipated skews to the ideological skew we really observe within the feeds of the bots. As in the principle textual content, we outline the ideological content material of a set of suggestions because the proportion of Republican-aligned movies minus the proportion of Democratic-aligned movies.

The metrics of curiosity embrace a number of video-level metrics, such because the variety of performs or views a video receives, in addition to its variety of shares, feedback and likes. Other than video-level metrics, we additionally gather channel-level metrics such because the cumulative variety of likes, followers and movies of a channel, in addition to whether or not the channel is verified or not. Provided that channel verification standing is a binary attribute (True or False), we apply a worth of 0.1 to channels that aren’t verified, and 0.9 to channels which can be verified. For robustness, we additionally take a look at all values between 0.5 and 1.0 in increments of 0.05 for verified channels (and inversely, 0.5 to 0 for unverified channels), and discover comparable outcomes (Supplementary Desk 17).

To retrieve predicted advice charges, for every week all through the period of the experiment we first isolate the set of movies seen by the bots throughout and earlier than the week in query. We then compute normalized values of the metric in query utilizing min–max normalization. Subsequent, we compute a bootstrapped measure of anticipated ideological content material by sampling n movies (the place n is equal to the variety of movies seen by a bot throughout a given week), weighted by the normalized metric in query. From every such pattern, we compute the proportion of Republican-aligned and Democratic-aligned movies and take the distinction to acquire the counterfactual ideological content material for that draw. This enables us to check the ideological content material of the sampled suggestions to the ideological content material really noticed within the feeds of the bots for a similar state–week–situation cells. The sampling course of is repeated 100 occasions for robustness to retrieve the imply and commonplace error of the anticipated ideological content material per week for every set of partisan bots.

Other than the single-value metrics talked about above, we compute three further engagement indices. The primary is a linear mixture of the normalized play, share, like and remark counts of a video. The second is a linear mixture of the cumulative likes, followers and movies of a channel, in addition to its verification standing. The third is a linear mixture of all video- and channel-level metrics. For every of those further metrics, the burden of every element is derived utilizing PCA, which identifies the linear mixture of elements that captures the utmost variance within the knowledge. We use the loadings (coefficients) from the primary principal element as weights, normalizing them to sum to 1, and repeat the sampling course of described above for every of those three mixed metrics.

The fashions that think about the variety of likes or feedback disregard the truth that these metrics usually are not impartial of the advice algorithm. Movies that obtain extra suggestions find yourself being considered extra, which in flip will increase their probability of receiving likes and/or feedback. Thus, to develop counterfactual fashions that seize the intrinsic ‘high quality’ of a video whereas partially isolating the impact of the algorithm, we normalize these metrics as follows. First, we take the variety of ‘suggestions’ a video receives because the variety of performs minus the variety of shares. This subtraction yields the variety of views acquired by the video, particularly by the advice algorithm, and never by customers sharing the video. With this suggestions metric, we moreover develop counterfactual fashions that account for the variety of likes or feedback per advice of a video.

Lastly, for every of the aforementioned metrics, we repeat the sampling processes whereas considering the potential favourability of latest content material. Particularly, we inversely scale every metric in query with the time between when an audit was performed and the publishing date of a given video, as soon as linearly and as soon as exponentially. In complete, this quantities to 39 totally different counterfactual fashions with which we examine noticed ideological content material and anticipated ideological content material. In Supplementary Fig. 1a, the leftmost level represents the noticed ideological skew within the suggestions of the bots (the proportion of Republican-aligned movies minus the proportion of Democratic-aligned ones). Every of the remaining factors presents the anticipated ideological skew as computed by the counterfactual fashions, with level markers indicating various recency-weighting schemes. If the noticed engagement metrics totally defined the partisan hole, we’d anticipate these engagement-based counterfactual skews to be near the noticed skew of roughly 0.2. As an alternative, throughout all 39 counterfactuals, the noticed skew in direction of Republican-aligned content material considerably exceeds the counterfactual skews, and lots of engagement-based fashions indicate a Democratic-leaning skew. Supplementary Desk 14 particulars the ideological skew computed by every mannequin, in addition to t-test outcomes evaluating the noticed and anticipated skews. Supplementary Tables 15 and 16 report analogous outcomes for fashions utilizing solely optimistic or solely unfavourable partisan movies, respectively.

Survey

We administered a preregistered survey with a pattern of 1,008 US-based TikTok customers to evaluate whether or not people had observed modifications to the content material of their TikTok feeds, notably political content material, over the previous yr. The survey was preregistered on OSF (https://osf.io/udywb/) and was deemed exempt by the institutional evaluate board of the authors (IRB protocol no. HRPP-2025-69).

Individuals had been recruited utilizing the web platform Prolific and screened to make sure they resided within the USA and had been lively customers of TikTok. The survey consisted of two elements: (1) a collection of open-ended text-entry questions; and (2) a collection of structured, scale-based questions. Open-ended objects requested members whether or not they had observed any modifications to the content material on their TikTok feed on the whole, any modifications to political content material particularly and whether or not the tone of political content material had turn out to be extra optimistic or unfavourable. Responses to those questions had been manually coded by the primary writer to find out whether or not members explicitly referenced modifications to political content material and, if that’s the case, whether or not they described seeing extra Republican-aligned or Democratic-aligned content material.

Structured questions requested members to charge, on a 0–10 scale, the extent to which their feed had shifted in direction of Democratic or Republican content material, turn out to be extra optimistic or unfavourable in tone, or featured extra political content material they agreed or disagreed with.

For every survey merchandise, we performed separate linear regression analyses that included participant political affiliation and demographic covariates (age, gender, race and schooling degree) as predictors. Full regression outcomes are introduced in Supplementary Tables 33 and 34 for the open-ended and structured questions, respectively. Participant demographic traits are summarized in Supplementary Desk 35, and the whole survey instrument is accessible in Supplementary Be aware 6.

Reporting abstract

Additional data on analysis design is accessible within the Nature Portfolio Reporting Abstract linked to this text.