Virtually two years in the past, we laid out our thought of recent metadata for the fashionable information stack, the Information Catalog 3.0.
Since then, it’s been a dream seeing our imaginative and prescient come to life. Lively metadata has develop into a class and central speaking level within the information world. Gartner, Forrester, and G2 alike made main strikes to scrap outdated concepts of information cataloging and introduce new ones round lively metadata and trendy cataloging.
This transformation was pushed by actual product developments, ones that turned this summary dream into one thing that information groups might really use.
In 2022, our engineering crew shipped over 200 new options, together with a model new model of our product. These enhancements proved the facility of lively metadata and even bought us ranked as a Chief within the Forrester Wave, Q2 2022!
Earlier than we tackle the following yr of recent options, let’s take a couple of minutes to look again on a tremendous yr. Listed here are 15 of our favourite options from 2022.
TL;DR: 15 updates in Atlan it is best to find out about
From siloed instruments to embedded context
We’ve all the time been pushed by one core thought — variety is the one actuality. Information groups are probably the most numerous groups at this time, from the instruments they use and the folks they work with to their very own use instances and personas. This variety is each a energy and a weak point.
Information groups work throughout a various set of instruments — BI instruments like Looker, collaboration instruments like GitHub, communication instruments like Slack, ticketing techniques like Jira, and extra. Too typically, these flip into siloed information techniques. In case you’re in a dashboard, you wish to know in case you can belief it then and there, reasonably than switching and looking throughout different instruments.
In 2022, we constructed variety into the core of our product. By activating metadata, Atlan sends essential context again into the workflows and instruments the place the people of information work on daily basis.
Improve collaboration and seize data with our Slack integration
With our Slack integration, launched in January 2022, we set the stage for in depth collaboration on the earth of metadata. Atlan’s deep integration with Slack goes past simply sending messages — it’s about sharing and capturing necessary data throughout your group.
Atlan’s Slack integration consists of:
- Wealthy preview hyperlinks: Share property along with your crew on Slack and see detailed previews.
- Workflow alerts: Get alerts in Slack at any time when an Atlan workflow fails or succeeds.
- Ask a query from Atlan: Ask a query by clicking on the Slack icon for any asset. That Slack dialog will even be documented robotically in Atlan for future reference.
- Doc threads immediately from Slack to Atlan: Simply share Slack conversations to particular property in Atlan.
- Search phrases and queries immediately from Slack: Seek for phrases and queries in Atlan with out leaving Slack.
Keep in move with our Atlan Chrome Extension
Lively metadata makes it potential for metadata to move effortlessly and shortly throughout your entire information stack, embedding enriched context and data in each device within the information stack.
Think about you’re a dashboard and wish to know who owns it and what a metric means. How would you go about it? Nicely, you’ll in all probability have to determine who the proprietor was. Then you definately would most probably take a display seize of the dashboard, e-mail or Slack it to that proprietor, and ask your questions. Not anymore!
Now you’ll be able to skip all these steps by merely clicking on the Atlan icon in your browser. It’ll open the Atlan Chrome extension proper in your dashboard, supplying you with all of the context you want proper once you want it. Atlan makes this potential within the instruments the place your information groups and information shoppers spend most of their time — Looker, Tableau On-line, Snowsight, Mode, Energy BI, BigQuery, and Salesforce, with extra instruments on the best way.
Remedy points quicker with our Jira integration
Monitoring points and fixing issues simply bought so much simpler with our Jira integration. For a lot of information groups, attending to the “root trigger” of damaged dashboards or inconsistent numbers generally is a lengthy, drawn-out course of. Not anymore!
With Atlan’s Jira integration, customers can now shortly open a brand new Jira ticket or handle an present ticket immediately from any Atlan asset. Simply click on the Atlan sidebar.
Reduce danger throughout information creation with our GitHub integration
When a knowledge engineer makes modifications to a dbt mannequin, they may unknowingly be affecting tens or a whole bunch of individuals, tables, or dashboards. How can we reduce this danger whereas additionally offering transparency for the information engineer and information shopper?
With our integration between Atlan and GitHub, information engineers can now entry all of the context they should reduce the chance for information shoppers.
Right here’s an instance of what this appears to be like like: say that you simply’re a knowledge engineer, and also you’ve created a pull request. When the GitHub motion runs, it is going to robotically create an inventory of all downstream property that will likely be affected by this request — earlier than you make the change. From there, you’ll be able to attain out to customers prematurely, or analysis and take a look at the property to see how they may be affected.
From dispersed, guide instruments to automated 360° context
Simply as information groups at this time use numerous instruments, their information ecosystem is made up of numerous information property. These are now not simply tables made up of rows and columns. Information property are assessments, metrics, pipelines, code, queries, dashboards, and extra — all scattered throughout totally different information instruments and techniques.
At present, once you onboard an engineer, you simply share a hyperlink to a GitHub repository. This has the code, documentation, historical past, collaborators, and every thing else you have to know concerning the code, multi functional place.
Nevertheless, it takes lots of time and guide effort to arrange and produce documentation into conventional information catalogs at this time — and simply as quickly as you’re completed, they’re old-fashioned. At Atlan, we needed to make information property simply as simple to share and perceive as software program on GitHub. Together with activating metadata again into day by day workflows, our workspace provides customers a 360-degree view of each asset by robotically pulling in and stitching collectively metadata from each device in your information stack.
Atlan Playbooks: the Zapier for metadata
One of many frequent questions we get from information groups is, “How can we automate our metadata?” Whereas different groups like advertising or gross sales can simply do action-based automation at scale with instruments like Zapier or Salesforce, information groups typically don’t have the bandwidth to code customized automations for every numerous use case. Why can’t there be a Zapier for metadata?
That’s the reason we’ve developed the primary low-code/no-code metadata automation for information groups. With Atlan’s Playbooks, customers can now create rule-based automations at scale. These can be utilized to drive infinite use instances throughout your group. Listed here are just a few examples:
- Deprecate property: Mark any property that haven’t been queried within the final 30 days as deprecated.
- Add/change possession: For Salesforce property which are lacking an proprietor, add RevOps because the proprietor.
- Report failed property: Publish an inventory on Slack of each desk and dashboard with a “failed” Airflow standing.
- Shield delicate information: Connect GDPR customized metadata on all property tagged as “PII”.
- Flag upstream alerts: Notify downstream homeowners when an upstream asset is tagged with a warning or announcement.
Scale back guide documentation with Trident recommendations
Conventional metadata administration generally is a tiresome, guide course of that requires lots of human intervention from information leaders and stewards.
That’s why we launched Trident, which offers automated recommendations for brand spanking new descriptions, homeowners, phrases, and classifications. Trident makes metadata enrichment simpler in order that information groups can spend extra time on significant initiatives and fewer time writing descriptions and classifying information. Since its launch, one-third of all description updates on Atlan have been made with Trident!
P.S. We lately gave a preview of the potential of Trident by combining it with GPT-3 to create Trident AI — more and more clever automated descriptions, enterprise phrases, READMEs, and extra.
Handle your information and lower prices with asset reputation and utilization metrics
One of many greatest challenges dealing with information groups at this time is managing their information and lowering information infrastructure prices. That’s why we launched utilization and recognition metrics.
Information groups can now analyze their Snowflake information property with out leaving Atlan. With these new metrics, organizations can determine their most used property, discover folks with probably the most context on property, determine unused property, and extra. These metrics embody:
- Asset question logs: The variety of occasions a knowledge asset has been queried (i.e. asset reputation), when it was final up to date, and many others.
- Person question logs: A knowledge asset’s high customers and most up-to-date customers, who final queried it, and many others.
- Price and efficiency optimization logs: Which queries are slowest, most costly, and many others.
Get full visibility and column-level lineage with our Fivetran Metadata API integration
Atlan’s deep partnership and integration with Fivetran creates the primary end-to-end view of the fashionable information stack. By leveraging Fivetran’s new Metadata API, Atlan now builds column-level lineage out of your Fivetran sources all the best way to BI instruments like Looker and Tableau — supplying you with full visibility throughout your trendy information stack with highly effective root trigger and impression evaluation.
We’re excited to accomplice with Atlan to convey higher metadata administration to our clients. Information governance is a big wrestle for information groups at this time.
Collectively, Fivetran and Atlan are fixing this downside by serving to information groups achieve much-needed visibility on the place information got here from, who accessed it, and what modifications have occurred within the pipeline.
Fraser Harris, VP of Product at Fivetran
Make metrics and fashions first-class residents with our dbt Semantic Layer integration
One other massive partnership from this yr was Atlan + dbt. With our new Semantic Layer integration, dbt metrics and fashions are actually first-class residents in Atlan.
Atlan’s deep integration with the Semantic Layer brings dbt’s wealthy metrics and fashions into the remainder of the information stack, permitting information groups to create repeatable, metadata properties — like desk homeowners and verified tags — and construct documentation requirements.
dbt’s metrics are additionally now part of Atlan’s column-level lineage, spanning from supply techniques and information storage to transformation and BI. With elevated context, visibility, and self-service, this integration helps numerous information groups work collectively higher.
Get a birds-eye view of your information property with our Reporting Middle
At Atlan, we’ve seen a sample of questions come up as our clients collect extra metadata. Our customers wish to perceive how their information panorama and its metadata are evolving.
That’s why we launched the Reporting Middle, which offers a fowl’s-eye view of all of the metadata updates in Atlan via visualizations that will help you monitor adoption, curation, and metadata administration. Briefly, Reporting Middle provides you information in your metadata. It doesn’t get extra meta than this.
With the Reporting Middle customers can now:
- Monitor their property
- Observe glossary phrases and classes
- Implement governance by monitoring personas, functions, classifications and requests
- Perceive queries and question runs from insights in Atlan.
From generic instruments to personalised experiences
One other problem with conventional information catalogs and metadata options is that they deal with each person the identical, delivering a generic expertise throughout the group.
The issue with that strategy is, context means various things to totally different folks. For instance, a knowledge engineer appears to be like at a desk and so they’re excited about transformations, information high quality, and pipeline stability. In the meantime, an analyst might care about column names and descriptions, frequency distributions, and well-liked queries ran towards that desk. Lastly, a enterprise person may simply see a metric and marvel “What does this metric imply?” and “Can I belief it?”
We needed to alter that narrative and construct a platform that delivers a customized expertise for every person throughout your group.
Create a Netflix-like information expertise with Personas and Functions
If Netflix can create a customized viewing expertise for his or her customers, we thought, why can’t Atlan do the identical factor for information groups? That’s precisely what we did with the introduction of personas and functions.
Personas and functions give admins the power to regulate entry to property, insurance policies, and preferences tailor-made to the position of every particular person. Personas present asset-level controls for every crew, whereas functions group property collectively for cross-team initiatives.
For instance, the advertising persona might give all advertising folks entry to sure Snowflake tables and Tableau property. With a function, some members of the advertising crew might also get entry to different advertising marketing campaign information and dashboards which are related to campaigns they’re operating this yr.
We’ve seen this having every kind of attention-grabbing implementations inside organizations. For example, many information groups utilizing Atlan have began eager about implementing paradigms like the information mesh, which at its core has this idea of a site. Utilizing personas and functions, you’ll be able to create extraordinarily personalised experiences by area, which then permits your information mesh to begin being profitable.
Curate person experiences with customized metadata
Customized metadata is likely one of the most useful options in Atlan. Together with creating their very own metadata (like a knowledge freshness rating or information high quality metrics), admins can choose which customized metadata ought to be seen to every persona whereas proscribing visibility to people who wouldn’t discover this info related.
For instance, it’s doubtless {that a} finance persona received’t want entry to dbt transformations and Airflow DAGs, however they might have to dig into lineage to troubleshoot an odd Salesforce report or question a Snowflake desk to find out ARR.
With customized metadata, Atlan offers the power to curate every individual’s distinctive expertise, serving to them work extra effectively with the precise info and guaranteeing they don’t get overwhelmed by your entire information property.
Energy personalised lively metadata with an all-new Atlan person interface
At Atlan, we’ve all the time been dedicated to serving to information groups work collectively higher. We began out as a knowledge crew ourselves and failed 3 times in making an attempt to implement a knowledge catalog. So we determined to construct Atlan to be the form of platform and accomplice that we wished we had again then.
After over a yr of labor, we launched a model new model of Atlan final yr. From a whole redesign of the Atlan person interface to highly effective lively metadata and personalization options, this replace made it simpler than ever for information groups to construct belief and democratize information with metadata.
From closed, siloed instruments to open techniques
When you will have a various set of customers, utilizing a various set of instruments and property, all throughout industries, information stacks, and extra, there will likely be a variety in use instances. Each firm’s use case is totally different, and one dimension doesn’t match all. That’s why we care about constructing open and extensible platforms that permit information groups to resolve their very own issues in their very own methods.
Drive infinite integrations and use instances with a totally open API structure
Offering a totally open API structure permits each motion to be programmatically pushed in Atlan. We now open all our metadata change occasions via AWS Eventbridge and a Kafka queue, Lamda webhooks, and extra.
On the core of the Atlan platform, every thing is open by default — and this enables information groups throughout any group to construct for and clear up all kinds of use instances. Listed here are a pair that we’ve seen organizations begin to implement.
Programmatic information archival and deletion
One among our clients, a media firm, purchases lots of exterior information. The info has an expiration date, so that they arrange guide processes to often handle and delete their expired information. This generally led to information not being deleted, leading to authorized and compliance dangers. This course of additionally pressured the information crew to often spend vital time on guide work, leading to misplaced productiveness and crew output.
Now, utilizing the Atlan API, a notification is triggered 30 days earlier than the contractual finish date for the information. This creates a Jira ticket, letting the suitable events know that the information can then be deleted or archived. By means of this new programmatic metadata workflow, the corporate was not solely capable of remove authorized danger but in addition save vital time and productiveness.
Lively information belief administration
One other buyer has built-in Atlan with a knowledge high quality device and energy lively information belief administration. They use our open APIs to usher in anomalies which are detected by the information high quality device, after which downstream BI property and probably impacted enterprise metrics are robotically flagged. For instance, if a desk feeding a KPI has a knowledge high quality difficulty, Atlan robotically creates an Announcement for the KPI within the Atlan enterprise glossary.
Construct customized event-based automations with our AWS EventBridge integration
Many groups throughout organizations, from information observability to incident administration, are constructing automations that may set off actions primarily based on occasions. However there’s by no means been something out there for metadata administration to create some of these event-driven use instances for metadata… till now.
Information groups can now use Atlan to create production-grade, event-driven automations for the world of metadata. This leverages our integration between Atlan and AWS EventBridge, an AWS service that creates an occasion and lets customers devour and construct use instances on it, to create Atlan metadata occasions into an EventBridge account.
Listed here are some examples of how this may work in Atlan:
- Possession alerts: Get notifications in Slack when there’s a change within the possession of an asset.
- Propagation and classification: If somebody marks a subject as PII in an upstream information supply, robotically create a masking coverage to alter all associated fields downstream.
- And numerous extra use instances, equivalent to notifications round schema modifications on property in Atlan for information engineering, login/logout occasions for safety groups, or triggering a Fivetran enrichment occasion to kick off an Atlan workflow.
—
💡 To study extra concerning the newest options in Atlan, take a look at shipped.atlan.com or our December product roundup. It’s also possible to subscribe to our Product Updates publication for all updates.
💡 To take a look at Atlan for your self, go to our product tour.
💡 Prepared to begin utilizing these new options? Attain out to our Gross sales crew or your Buyer Success Supervisor to learn how.