Atlassian Dramatically Improves Machine Learning Model Accuracy & Deployment Times

Atlassian aims to provide exceptional experiences to its vast base of daily users. In support of that objective, the Search and Smarts team applies predictive analytics and Machine Learning to power intelligent experiences across its customer ecosystem.

Before engaging with Tecton, one of the Search and Smarts team’s most visible projects was to predict and recommend user mentions in Jira and Confluence. Data scientists spent multiple months building the first features and generating training datasets with promising results. But they soon realized that deploying these features in production would be a challenge, particularly due to the use of streaming data.

To serve their features in production, the team set out to build an internal feature store. They assigned ~3 engineers who successfully built the first version in about one year. The internal feature store addressed the most pressing issue of serving features online, but didn’t bridge the data science / engineering gap and would not scale to support all of Atlassian’s needs. Teams were still operating in silos, it still took months of effort to build production pipelines, and the lack of training / serving parity was reducing the accuracy of predictions.

To address these gaps, Atlassian evaluated the Tecton feature store to manage the complete lifecycle of features in Atlassian’s operational ML stack. After a successful Proof of Concept, Tecton was deployed in production to power predictive experiences, such as recommending content fields in Jira and Confluence. By using Tecton as their data platform for ML, Atlassian has accelerated the delivery of new features from a range of 1-3 months to just 1 day, and has increased the prediction accuracy of existing models by 2%. These changes are directly improving over 200,000 customer interactions every day in Jira and Confluence.

ML Use Case Priorities

The Search and Smarts team builds and operates multiple ML applications to improve the customer experience in Atlassian products. For example, the team built an application to predict user mentions, assignees, and labels in Jira and Confluence. User mentions are used to draw someone’s attention to a page or comment, while assignees and labels are used to allocate work and categorize content. Instead of simply providing a dropdown list by alphabetical order, models predict and suggest the right content up to 85% of the time. For power users that are assigning 100s of tickets a day, this can meaningfully reduce the time spent on repetitive tasks.

Another high-priority use case is to improve the in-product search experience. Search is used extensively to quickly find relevant pages and to track down issues. It’s an essential capability for large teams that may have thousands of pages and issues to search through.

Challenges Getting ML to Production

The Search and Smarts team quickly realized that building and deploying these ML applications at scale would require a new platform for operational ML. Indeed, Atlassian operates at massive scale:

• >174,000 customers
• Billions of events generated every day
• Hundreds of millions of feature-key combinations per model
• >1M individual predictions generated every day

Getting the first models to production at this level of scale was a complex process. Without a feature store in place to manage the lifecycle of ML features, teams were struggling with:

Lack of collaboration tools spanning data science and engineering: The teams didn’t have a single repository of features that spanned both development and production environments. Keeping code in sync between data science and engineering required extensive manual coordination through meetings and written documents.

Complexity of incorporating streaming data: Many of Atlassian’s features are built with a combination of batch and streaming data to leverage the company’s valuable historical information while also capturing the latest events. The use of streaming data introduced unique challenges, such as carefully managing eventtime to prevent data leakage in training datasets and implementing streaming pipelines in production.

Lead times of multiple months to deploy production pipelines: Features created by data scientists were not production-ready. Much of the data scientists’ work needed to be reimplemented for production by a separate data engineering team. This included creating and maintaining the infrastructure to process features with streaming data, hardening production pipelines, and adding monitoring to batch pipelines.

Reduced prediction accuracy due to training / serving skew: The pipelines used to serve online and offline data were built with different code bases and different data sources. This introduced online / offline data skew which in turn reduced the accuracy of predictions.

To overcome these challenges, the team set out to build an internal feature store from scratch. Inspired by the Uber Michelangelo platform, Atlassian’s original feature store was successfully delivered by a small team of ~3 full-time engineers in one year. The feature store was successful in serving features online, but it was not designed to solve the tooling gap between data science and engineering. Data scientists were still building features in a separate environment from production, and pipelines still needed to be re-implemented in production.

Solution

Tecton was founded by the creators of Uber Michelangelo and provides an enterprise-ready feature store. Atlassian believed that Tecton could address many of its challenges in productionizing ML, while eliminating the need to manage and maintain its own internal feature store.

Atlassian and Tecton ran a Proof Of Concept project. Over the course of a few months, the teams worked together to implement the Tecton enterprise feature store and test the most important use case on Tecton: the models that predict user mentions, assignees, and labels in Jira and Confluence. After the successful POC, Atlassian is now running multiple models in production on Tecton, and has decided to deploy the Tecton feature store as a core component of Atlassian’s ML stack for future use cases.

Results