At Orfium, our mission is to unlock the full potential of content. That’s why we created Sync-Tracker, a groundbreaking solution for managing licensed and unlicensed music on YouTube. Whether you’re a music production company, a content creator, or simply someone uploading content on YouTube, Sync-Tracker is designed to cater to your needs. With Sync-Tracker, you can easily handle licensed and unlicensed music, while providing music production companies a platform to sell licenses to content creators. Additionally, users can enforce their licenses through YouTube’s Content ID, making Sync-Tracker a powerful tool for protecting your rights and ensuring compliance.
SyncTracker is a powerful licensing software that keeps a close eye on YouTube claims generated after the upload of a video, regardless of its privacy settings. Leveraging advanced audio and melody-matching algorithms, SyncTracker analyzes the content owner’s extensive catalog of assets to identify any potential matches. For more information on YouTube’s claims and policies, please refer to the detailed documentation available here.
Cracking the Code: Insights into Efficient Claim Tracking đ§âđ»
To track claims efficiently, our technical approach involves leveraging the ClaimSearch.list API call. This API call allows us to retrieve claims based on specific statuses in a paginated manner. For instance, if there are 100 claims on YouTube, the initial request may only fetch 20-30 claims, accompanied by a nextPageToken. To access claims from subsequent pages, additional requests with the nextPageToken are necessary. This iterative process of making sequential ClaimSearch.list API calls enables us to gather all the claims seamlessly. Below, you’ll find a high-level diagram outlining the flow of tracking and processing claims, reflecting our recent advancements in this area.
What’s the daily maximum number of claims we can track from YouTube content ID?Â
Determining the maximum number of claims we can track from YouTube’s Content ID per day involves calculating the theoretical upper limit based on response time and claim quantity. Assuming an average of 0.5 seconds for each request/response and approximately 20 claims obtained per call, we can estimate a capacity of around 3.5 million claims per day. Previously, we faced limitations in approaching this limit due to the sequential processing of each claim batch.
The Chronicle of the Ages: Unveiling the History of Traffic Increaseâł
The solution we implemented until the end of 2022 served us flawlessly, catering to the manageable volume of claims we handled at the time. By the close of 2022, Orfium’s YouTube snapshot encompassed approximately 250,000 assets, with YouTube generating an average of around 30,000 claims per day for these assets.
However, the landscape changed in early 2023, when Orfium secured significant contracts with large Music Production Companies (MPCs). This development projected a substantial 5.5x increase in the Content ID (CiD) size, resulting in a total of 1.4 million assets. With this expansion, the claim traffic was also expected to grow. Estimating the precise increase in claim traffic proved challenging as it does not have a linear relationship with the number of assets. The quantity of claims generated depends on the popularity of each song, not just the sheer number of songs. We encountered scenarios where a vast catalog generated regular traffic due to the songs’ lack of popularity and vice versa. Consequently, we anticipated a claim traffic growth of at least 3-4x. However, our old system could only handle a maximum increase of 2x traffic đŁ, prompting the need for upgrades and improvements.
Success Blueprint: Strategizing for Achievement and Risk Mitigation đ§
As our development team identified the impending performance bottleneck, we immediately recognized the need for proactive measures to prevent system strain and safeguard our new contracts. The urgency of the situation was acknowledged by the product Business Unit (BU), prompting swift action planning.
We initiated a series of brainstorming meetings to explore potential short-term and long-term solutions. With the adage “prevention is better than cure” guiding our approach, we devised a backlog of mitigation strategies to address the increased risk until the customer deals were officially signed. We were given a demanding timeframe of 4-6 weeks to complete all activities, including brainstorming meetings, analysis, design, implementation, testing, production delivery, and monitoring. The challenge lay in the uncertainty surrounding the effectiveness of the chosen solution.
After several productive brainstorming sessions, we compiled a comprehensive list of potential alternatives. These ranged from pure engineering decisions to YouTube-side configuration ideas, ensuring that every possible avenue was explored to ensure the safety and security of the new contracts.
Unleashing the Power of Solutions: Overcoming Traffic Challenges and Streamlining Claims Tracking đ§©
With the primary goal of reducing the time spent in claims tracking, we embarked on a journey to introduce innovative solutions. One such solution involved the implementation of a new metric in Datadog, designed to detect any missing claims. This introduced a proactive approach to monitor if any claims were not being tracked by our application and enabled us to identify and address related issues promptly. By leveraging this powerful tool, we gained valuable insights into potential issues such as increased traffic and timeouts, empowering us to take swift action and maintain optimal performance.
The solutions we discussed to enhance our claims tracking process included:
- Exclude claims from processing: Through our analysis, we discovered that approximately 70% of the claims tracked from YouTube were generated for private videos that were unlikely to be made public. Recognizing this, we proposed a solution to exclude these videos from our main processing unit and manage them separately. By doing so, we could reduce the volume of claims processed within the same timeframe by approximately 70%. This exclusion resulted in a significant 20% reduction in the end-to-end (E2E) processing time.
- Asynchronous claim process flow: To address the long-term scalability of our system, we aimed to redesign SyncTracker’s main flow to support asynchronous operation. This change showed promising potential as it would allow us to process each claim batch from YouTube concurrently, enabling horizontal scaling through multiple workers. To ensure the sufficiency of this solution, we decided to implement a feature flag. This feature flag would provide the flexibility to switch between asynchronous and synchronous execution dynamically, preserving the system’s existing operation and quickly addressing any potential issues that may arise..
Solving the Puzzle: Addressing Production System Issues through Troubleshooting đ
In the world of production systems, Murphy’s Law always lurks, ready to present unexpected challenges. After deploying the async implementation, everything seemed to be working smoothly for approximately 8 hours. However, we soon encountered a significant increase in the processing time of each request, disrupting the expected flow. Fortunately, thanks to our vigilant monitoring through Datadog metrics, we were able to detect the issue before it caused any service disruptions.
In light of the async solution implementation and concurrent query execution, our workflow encountered a significant setback due to time-intensive database queries for each monitored claim. This resulted in multiple query locks and idle worker processes. To tackle this problem, we strategically tackled the situation by identifying and eliminating unnecessary indexes from the license table. Additionally, we meticulously benchmarked the pertinent license-matching queries. Remarkably, we discovered that the queries filtered by video_id were the slowest.
In order to effectively address this performance bottleneck, we took proactive measures by creating a brand-new index specifically designed for license-matching queries. This optimization led to a remarkable reduction in overall processing time, with the median duration plummeting from 4-5 minutes to an impressive 1 minute. This represents a substantial decrease of 400-500%, greatly enhancing efficiency and productivity.
Closing the Chapter: Onboarding Outcomes Explored đ
Following the successful onboarding of new customers, we witnessed a remarkable surge in claim traffic, experiencing an eightfold increase from 50,000 to nearly 400,000.Â
Presently, the SyncTracker application maintains exceptional stability and seamlessly manages the influx of claim traffic. Based on our careful projections, we have determined that our system is fully capable of accommodating claim traffic up to the maximum threshold allowed by the YouTube API, which amounts to approximately 3 million claims per day. Furthermore, we are proud to report a remarkable decrease of approximately 700% in the total number of missing claims. This figure has significantly dropped from 1,500 down to less than 100, underscoring the effectiveness and efficiency of our platform.
Considering all factors, there are additional minor optimizations that are currently being developed and are expected to yield an average time-saving of 15%-20%. These enhancements are still in the pipeline and have not yet been implemented in the production environment. It is important to note that despite our current success, this milestone does not signify the end of our journey. We remain committed to continuously improving our processes and striving for further advancements.
Abbreviations đ€
MPC – Music Production Companies
CiD – YT Content ID
CMS – Content Management System -used interchangeably with CiD-
YT – YouTube
ST – SyncTracker
BU – Business Unit
E2E – End to End
