All Things Engineering
Engineering & Developer Experience from the best in class. Culture, Architecture, DevOps and stuff at scale.
🏱 %company% Engineering
AirBnb Engineering
Last posted 1 week, 1 day ago
Upgrading Data Warehouse Infrastructure at Airbnb
Upgrading Data Warehouse Infrastructure at Airbnb Upgrading Data Warehouse Infrastructure at Airbnb

This blog aims to introduce Airbnb’s experience upgrading Data Warehouse infrastructure to Spark and Iceberg.By: Ronnie Zhu, Edgar Rodriguez, Jason Xu, Gustavo Torres, Kerim Oktay, Xu ZhangIntroductionIn this blog, we will introduce our motivations for upgrading our Data Warehouse Infrastructure to Spark 3 and Iceberg. We will briefly describe the current state of Airbnb data warehouse infrastructure and the challenges. We will then share our learnings from upgrading one critical production workload: event data ingestion. Finally, we will share the results and the lessons learned.ContextAirbnb’s Data Warehouse (DW) storage was previously migrated from legacy HDFS clusters to S3 to provide b


1 week, 1 day ago @ medium.com
How Airbnb safeguards changes in production
How Airbnb safeguards changes in production How Airbnb safeguards changes in production

Part II: Near Real-time ExperimentsBy: Mike Lin, Preeti Ramasamy, Toby Mao, Zack Loebel-BegelmanIn our first post we discussed the need for a near real time Safe Deploy system and some of the statistics that power its decisions. In this post we will cover the architecture and engineering choices behind the various components that Safe Deploys comprises.Designing a near real-time experimentation system required making explicit tradeoffs among speed, precision, cost, and resiliency. An early decision was to limit near real-time results to only the first 24 hours of an experiment — enough time to catch any major issues and transition to using comprehensive results from the batch pipeline. The 


4 weeks ago @ medium.com
My Journey to Airbnb — Veerabahu Chandran
My Journey to Airbnb — Veerabahu Chandran My Journey to Airbnb — Veerabahu Chandran

My Journey to Airbnb — Veerabahu ChandranLearning and growing in Airbnb’s new Bangalore Tech CenterVeera Chandran is an engineer in Airbnb’s new Bangalore Tech Center, where his team builds out technical systems to support hosts. As a lifelong learner, he has a passion for exploring new technologies and diving into practical problems. He’s excited to be tackling both the technical challenges of building new architecture and the organizational challenges of building out the capabilities of a new office.Here’s Veera’s story:Learning and exploringI grew up in Tamil Nadu, in the South of India. I was always a curious kid, trying to understand how everything worked, so when it came to choosing a


1 month, 2 weeks ago @ medium.com
Sisyphus and the CVE Feed: Vulnerability Management at Scale
Sisyphus and the CVE Feed: Vulnerability Management at Scale Sisyphus and the CVE Feed: Vulnerability Management at Scale

AuthorsKeziah Perez Sonder Plattner, Senior Software EngineerKadia Mashal, Engineering ManagerIntroductionEvery engineer knows that security is a never-ending problem. Until we delete all our code and move into a cottage in the woods, we have to accept that there is no such thing as 100% secure software. You could be doing everything perfectly, and a publicly known vulnerability (CVE) could emerge for the most updated version of a third party library in your infrastructure. Things are secure until they are not. Like with Sisyphus, the boulder will never reach the top of the hill.Rather than eliminating vulnerabilities, the goal of a vulnerability management program should be to quickly and 


1 month, 3 weeks ago @ medium.com
Airbnb’s Approach to Access Management at Scale
Airbnb’s Approach to Access Management at Scale Airbnb’s Approach to Access Management at Scale

How Airbnb securely manages permissions for our large team of employees, contractors, and call center staff.By: Paul BramsenIntroductionAirbnb is a company that is built on trust. An important piece of this trust comes from protecting the data that our guests and hosts have shared with us. One of the ways we do this is by following the principle of least privilege. Least privilege dictates that–in an ideal world–an employee has the exact permissions they need at the moment their job requires them. Nothing more, nothing less. Anything more introduces unnecessary risk–whether from a malicious employee, compromised laptop, or even just an honest mistake. Anything less inhibits productivity.Not


1 month, 3 weeks ago @ medium.com
Incident Management
Incident Management Incident Management

Automated Incident Management Through SlackHow Airbnb automates incident management in a world of complex, rapidly evolving ensemble of microservices.Vlad VassilioukIncident ManagementIncidents are unforeseeable events that disrupt normal business operations and are inevitable in complex systems that must be up and running 24/7. This is why it’s important to prepare and to train people to handle incidents in a timely and organized manner. Although each incident is unique, we follow the same procedure for detection, escalation, management, and resolution of incidents.At Airbnb, we utilize a service oriented infrastructure which involves many interconnected services managed by small teams. Qu


2 months, 1 week ago @ medium.com
My Journey to Airbnb — Beti Gathegi
My Journey to Airbnb — Beti Gathegi My Journey to Airbnb — Beti Gathegi

My Journey to Airbnb — Beti GathegiFrom exploring careers across continents to now helping others find their place at Airbnb.After trying a series of careers ranging from television production to university communications and marketing, Beti Gathegi works as a Senior Program Manager on the TechED (technical education) team at Airbnb. When she’s not lurking in the #bookworms Airbnb Slack channel, you can find Beti leading Bootcamp, our onboarding program for new technical hires, which takes engineers and data scientists through their first commit at Airbnb. Before this role, Beti was a recruiting program manager for Connect, Airbnb’s engineering apprenticeship program targeted at people from


2 months, 2 weeks ago @ medium.com
How Airbnb Safeguards Changes in Production
How Airbnb Safeguards Changes in Production How Airbnb Safeguards Changes in Production

Part I: Evolution of Airbnb’s experimentation platformBy: Michael Lin, Toby Mao, Zack Loebel-BegelmanIntroductionAs Airbnb has grown to a company with over 1,200 developers, the number of platforms and channels for pushing changes to our product — and the number of daily changes we push into production — has also grown tremendously. In the face of this growth, we constantly need to scale our ability to detect errors before they reach production. However, errors inevitably slip past pre-production validation, so we also invest heavily in mechanisms to detect errors quickly when they do make it to production. In this blog post we will cover the motivations and foundations for a system for saf


2 months, 3 weeks ago @ medium.com
T-LEAF: Taxonomy Learning and EvaluAtion Framework
T-LEAF: Taxonomy Learning and EvaluAtion Framework T-LEAF: Taxonomy Learning and EvaluAtion Framework

How we applied qualitative learning, human labeling and machine learning to iteratively develop Airbnb’s Community Support Taxonomy.By: Mia Zhao, Peggy Shao, Maggie Hanson, Peng Wang, Bo ZengBackgroundTaxonomies are knowledge organization systems used to classify and organize information. Taxonomies use words to describe things — as opposed to numbers or symbols — and hierarchies to group things into categories. The structure of a taxonomy expresses how those things relate to each other. For instance, a Superhost is a type of Host and a Host is a type of Airbnb User. Taxonomies provide vital terminology control and enable downstream systems to navigate information and analyze consistent, st


3 months, 1 week ago @ medium.com
Airbnb’s Trip to Linaria
Airbnb’s Trip to Linaria Airbnb’s Trip to Linaria

Learn how Linaria, Airbnb’s newest choice for web styling, improved both developer experience and web performanceCSS is a critical component of every web application, and many solutions have evolved for how styles are written by developers and delivered to visitors. In this post we’ll take you through Airbnb’s journey from Sass to CSS-in-JS and show you why we landed on Linaria, a zero-runtime CSS-in-JS library, and the impact it has had on the developer experience and performance of Airbnb’s web app.From Sass to CSS-in-JSIn 2016, our web frontend was in a monolithic Ruby on Rails app using a combination of Sprockets, Browserify, and Sass. We had a Bootstrap-inspired internal toolkit for st


3 months, 2 weeks ago @ medium.com
Netflix Engineering Netflix Engineering
Last posted 6 days, 13 hours ago
Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support
 Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support


Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloadsby Kostas ChristidisIntroductionTimestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos, our media encoding platform. Over the past 2.5 years, its usage has increased, and Timestone is now also the priority queueing engine backing Conductor, our general-purpose workflow orchestration engine, and BDP Scheduler, the scheduler for large-scale data pipelines. All in all, millions of critical workflows within Netflix now flow through Timestone on a daily basis.Timestone clients can create queues, enqueue mes


6 days, 13 hours ago @ netflixtechblog.com
Reinforcement Learning for Budget Constrained Recommendations
Reinforcement Learning for Budget Constrained Recommendations Reinforcement Learning for Budget Constrained Recommendations

by Ehtsham Elahiwith James McInerney, Nathan Kallus, Dario Garcia Garcia and Justin BasilicoIntroductionThis writeup is about using reinforcement learning to construct an optimal list of recommendations when the user has a finite time budget to make a decision from the list of recommendations. Working within the time budget introduces an extra resource constraint for the recommender system. It is similar to many other decision problems (for e.g. in economics and operations research) where the entity making the decision has to find tradeoffs in the face of finite resources and multiple (possibly conflicting) objectives. Although time is the most important and finite resource, we think that i


1 month, 1 week ago @ netflixtechblog.com
Virtual Production — A Validation Framework For Unreal Engine
Virtual Production — A Validation Framework For Unreal Engine Virtual Production — A Validation Framework For Unreal Engine

Virtual Production — A Validation Framework For Unreal EngineBy Adam Davis, Jimmy Fusil, Bhanu Srikanth and Girish BalakrishnanGame Engines in Virtual ProductionThe use of Virtual Production and real time technologies has markedly accelerated in the past few years. At Netflix, we are always thrilled to see technology enable new ways of telling stories, and the use of these techniques on some of our shows like 1899 and Super Giant Robot Brothers has given us a front row seat to this exciting evolution in filmmaking. Each production that deploys these methods is an opportunity for the crew, tech manufacturers and us–the Netflix Production Innovation team–to learn, innovate and collaborate tow


1 month, 3 weeks ago @ netflixtechblog.com
Data Mesh — A Data Movement and Processing Platform @ Netflix
Data Mesh — A Data Movement and Processing Platform @ Netflix Data Mesh — A Data Movement and Processing Platform @ Netflix

Data Mesh — A Data Movement and Processing Platform @ NetflixBy Bo Lei, Guilherme Pires, James Shao, Kasturi Chatterjee, Sujay Jain, Vlad SydorenkoBackgroundRealtime processing technologies (A.K.A stream processing) is one of the key factors that enable Netflix to maintain its leading position in the competition of entertaining our users. Our previous generation of streaming pipeline solution Keystone has a proven track record of serving multiple of our key business needs. However, as we expand our offerings and try out new ideas, there’s a growing need to unlock other emerging use cases that were not yet covered by Keystone. After evaluating the options, the team has decided to create Data


2 months ago @ netflixtechblog.com
Formulating ‘Out of Memory Kill’ Prediction on the Netflix App as a Machine Learning Problem
Formulating ‘Out of Memory Kill’ Prediction on the Netflix App as a Machine Learning Problem Formulating ‘Out of Memory Kill’ Prediction on the Netflix App as a Machine Learning Problem

by Aryan Mehrawith Farnaz Karimdady Sharifabad, Prasanna Vijayanathan, Chaïna Wade, Vishal Sharma and Mike SchassbergerAim and Purpose — Problem StatementThe purpose of this article is to give insights into analyzing and predicting “out of memory” or OOM kills on the Netflix App. Unlike strong compute devices, TVs and set top boxes usually have stronger memory constraints. More importantly, the low resource availability or “out of memory” scenario is one of the common reasons for crashes/kills. We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. With large 


2 months, 2 weeks ago @ netflixtechblog.com
How Netflix Content Engineering makes a federated graph searchable (Part 2)
How Netflix Content Engineering makes a federated graph searchable (Part 2) How Netflix Content Engineering makes a federated graph searchable (Part 2)

By Alex Hutter, Falguni Jhaveri, and Senthil SayeebabaIn a previous post, we described the indexing architecture of Studio Search and how we scaled the architecture by building a config-driven self-service platform that allowed teams in Content Engineering to spin up search indices easily.This post will discuss how Studio Search supports querying the data available in these indices.Data consumption from Studio Search DGSIntroductionWhen we say Content Engineering teams are interested in searching against the federated graph, the use-case is mainly focused on known-item search (a user has an item or items in mind they are trying to view or navigate to but need to use an external information 


3 months, 3 weeks ago @ netflixtechblog.com
Scaling Appsec at Netflix (Part 2)
Scaling Appsec at Netflix (Part 2) Scaling Appsec at Netflix (Part 2)

By Astha Singhal, Lakshmi Sudheer, Julia KnechtThe Application Security teams at Netflix are responsible for securing the software footprint that we create to run the Netflix product, the Netflix studio, and the business. Our customers are product and engineering teams at Netflix that build these software services and platforms. The Netflix cultural values of ‘Context not Control’ and ‘Freedom and Responsibility’ strongly influence how we do Security at Netflix. Our goal is to manage security risks to Netflix via clear, opinionated security guidance, and by providing risk context to Netflix engineering teams to make pragmatic risk decisions at scale.A few years ago, we published this blog p


4 months ago @ netflixtechblog.com
A Survey of Causal Inference Applications at Netflix
A Survey of Causal Inference Applications at Netflix A Survey of Causal Inference Applications at Netflix

At Netflix, we want to entertain the world through creating engaging content and helping members discover the titles they will love. Key to that is understanding causal effects that connect changes we make in the product to indicators of member joy.To measure causal effects we rely heavily on AB testing, but we also leverage quasi-experimentation in cases where AB testing is limited. Many scientists across Netflix have contributed to the way that Netflix analyzes these causal effects.To celebrate that impact and learn from each other, Netflix scientists recently came together for an internal Causal Inference and Experimentation Summit. The weeklong conference brought speakers from across th


4 months, 2 weeks ago @ netflixtechblog.com
Evolution of ML Fact Store
Evolution of ML Fact Store Evolution of ML Fact Store

by Vivek KaushalAt Netflix, we aim to provide recommendations that match our members’ interests. To achieve this, we rely on Machine Learning (ML) algorithms. ML algorithms can be only as good as the data that we provide to it. This post will focus on the large volume of high-quality data stored in Axion — our fact store that is leveraged to compute ML features offline. We built Axion primarily to remove any training-serving skew and make offline experimentation faster. We will share how its design has evolved over the years and the lessons learned while building it.TerminologyAxion fact store is part of our Machine Learning Platform, the platform that serves machine learning needs across N


5 months, 1 week ago @ netflixtechblog.com
How Netflix Content Engineering makes a federated graph searchable
How Netflix Content Engineering makes a federated graph searchable How Netflix Content Engineering makes a federated graph searchable

By Alex Hutter, Falguni Jhaveri and Senthil SayeebabaOver the past few years Content Engineering at Netflix has been transitioning many of its services to use a federated GraphQL platform. GraphQL federation enables domain teams to independently build and operate their own Domain Graph Services (DGS) and, at the same time, connect their domain with other domains in a unified GraphQL schema exposed by a federated gateway.As an example, let’s examine three core entities of the graph, each owned by separate engineering teams:Movie: At Netflix, we make titles (shows, films, shorts etc.). For simplicity, let’s assume each title is a Movie object.Production: Each Movie is associated with a Studio


5 months, 3 weeks ago @ netflixtechblog.com
Pinterest Engineering
Last posted 1 month, 2 weeks ago
Online Data Migration from HBase to TiDB with Zero Downtime
Online Data Migration from HBase to TiDB with Zero Downtime Online Data Migration from HBase to TiDB with Zero Downtime

Ankita Girish Wagh | Senior Software Engineer, Storage and CachingIntroduction and MotivationAt Pinterest, HBase is one of the most critical storage backends, powering many online storage services like Zen (graph database), UMS (wide column datastore), and Ixia (near real time secondary indexing service). The HBase Ecosystem, though having various advantages like strong consistency at row level in high volume requests, flexible schema, low latency access to data, Hadoop integration, etc. cannot serve the needs of our clients for the next 3–5 years. This is due to high operational cost, excessive complexity, and missing functionalities like secondary indexes, support for transactions, etc.Af


1 month, 2 weeks ago @ medium.com
Debugging Ad Delivery At Pinterest
Debugging Ad Delivery At Pinterest Debugging Ad Delivery At Pinterest

Nishant Roy | Engineering Manager, Ads Serving PlatformIntro & BackgroundThe Pinterest ads serving platform delivered >$2.5 billion in ad spend in 2021 from thousands of advertisers. Our customer operations team receives 600+ tickets on average every month from advertisers who are looking to understand their performance on our platform. One of the most common questions we receive is why a particular advertiser/ad campaign is not fully utilizing its budget. This question requires a deep analysis of an ad recommendation system consisting of 5+ microservices, 1M+ lines of code, and 100+ active developers, serving >90 million requests everyday. This blog describes how we built a system to swift


3 months, 1 week ago @ medium.com
Estimating Potential Audience Size of an Ad at Pinterest
Estimating Potential Audience Size of an Ad at Pinterest Estimating Potential Audience Size of an Ad at Pinterest

Chanheum (Sean) Cho | ML Engineer, Ads Intelligence; Ruixin Qiang |ML Engineer, Ads Intelligence; Keshava Subramanya |Engineering Manager, Ads IntelligenceIntroductionUnderstanding the size of the potential audience of an ad is an important consideration for an advertiser. It enables advertisers to estimate the total population who might be interested in the products or services they advertise and plan their budgets ahead of time. The Ads Intelligence team at Pinterest provides a service called Potential Audience Size in the Ads Manager, so the advertisers can understand their target audience size while they configure their ad groups. The service updates the estimate in real time as the aud


4 months ago @ medium.com
Improving Distributed Caching Performance and Efficiency at Pinterest
Improving Distributed Caching Performance and Efficiency at Pinterest Improving Distributed Caching Performance and Efficiency at Pinterest

Kevin Lin | Software Engineer, Storage and CachingIntroductionPinterest’s distributed caching system, built on top of open source technologies memcached and mcrouter, is a critical component of the production infrastructure stack. Pinterest’s cache-as-a-service platform is responsible for driving down application latency across the board, reducing the overall cloud cost footprint, and ensuring adherence to strict sitewide availability targets.Today, Pinterest’s memcached fleet spans over 5000 EC2 instances across a variety of instance types optimized along compute, memory, and storage dimensions. Collectively, the fleet serves up to ~180 million requests per second and ~220 GB/s of network 


4 months, 3 weeks ago @ medium.com
Manas HNSW Streaming Filters
Manas HNSW Streaming Filters Manas HNSW Streaming Filters

Tim Koh | Software Engineer, Core Product Serving Infra; George Wu | (former) Software Engineer, Core Product Serving InfraIntroductionEmbedding-based retrieval is a core center piece of our recommendations engine at Pinterest. We support a myriad of use cases, from retrieval based on content similarity to learned retrieval. It’s powered by our in-house search engine — Manas — which provides Approximate Nearest Neighbor (ANN) search as a service, primarily using Hierarchical Navigable Small World graphs (HNSW).While traditional token-based search retrieves documents on term matching on a tree of terms with logical connectives like ANDs and ORs, ANN search retrieves based on embedding simila


5 months ago @ medium.com
NVIDIA 2022 Talk: Evolution of Web-Scale Engagement Modeling at Pinterest
NVIDIA 2022 Talk: Evolution of Web-Scale Engagement Modeling at Pinterest NVIDIA 2022 Talk: Evolution of Web-Scale Engagement Modeling at Pinterest

Prabhat Agarwal | Research ScientistWatch Prabhat Agarwal’s talk from the NVIDIA GTC Conference on March 23, 2022.https://medium.com/media/c160a4c48af4ff09c6ed83e10d26a5e8/hrefTo learn more about engineering at Pinterest, check out the rest of our Engineering Blog, and visit our Pinterest Labs site. To view and apply to open opportunities, visit our Careers page.NVIDIA 2022 Talk: Evolution of Web-Scale Engagement Modeling at Pinterest was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

5 months, 3 weeks ago @ medium.com
Presentamos PinFlex: el modelo de Pinterest para el futuro en el trabajo
Presentamos PinFlex: el modelo de Pinterest para el futuro en el trabajo Presentamos PinFlex: el modelo de Pinterest para el futuro en el trabajo

Jeremy King | Sr Vice Presidente, ingenierĂ­aThis article was originally published in English. Read the English version here.Hace dos años, estaba terminando mi primer año en Pinterest como Vicepresidente SĂ©nior. Nuestras oficinas cerraron y el trabajo y la vida, como los conocĂ­amos, cambiaron de golpe. Al adaptarnos a los cambios drĂĄsticos a nuestro alrededor, aprendimos mucho sobre el trabajo y sobre lo que no era un lugar de trabajo. Los beneficios del trabajo flexible y nuestra capacidad de ser productivos desde cualquier lugar dieron muchas ventajas a nuestros empleos, ya que se eliminaron largos viajes al trabajo y pudimos estar mucho mĂĄs presentes en los momentos importantes de nuestr


5 months, 3 weeks ago @ medium.com
How Pinterest built its Trust & Safety team
How Pinterest built its Trust & Safety team How Pinterest built its Trust & Safety team

Maisy Samuelson | Head of Trust & Safety Product“Inch by inch, row by row, I’m gonna make this garden grow” — Garden Song by Peter, Paul & MarySince I started working on Trust & Safety four years ago, the team has grown quickly and learned a lot about how to protect Pinners from spam and account takeovers as well as unsafe content like nudity, self-harm, hate speech, and harassment.Back in 2017, the Trust & Safety team (like Pinterest itself) was small and spent a lot of time fighting attacks. During attacks we scrambled to figure out product nuances, where and how data was stored, wait for queries to run, write scripts to update production databases and fix issues in user accounts, all und


6 months ago @ medium.com
Introducing PinFlex: Pinterest’s model for the Future of Work
Introducing PinFlex: Pinterest’s model for the Future of Work Introducing PinFlex: Pinterest’s model for the Future of Work

Jeremy King | Senior Vice President, EngineeringTwo years ago, as I was closing in on my first annual “Pinniversary” as the Chief of Engineering at Pinterest, our offices closed, and work (read: life) as we knew it changed abruptly. As we all adapted to the drastic changes around us, we learned a lot about what was working and what wasn’t in the workplace. The benefits of working flexibly and our ability to be productive from anywhere empowered us to do our jobs well while eliminating long commutes and allowing us to be more present than ever before for important moments with family. We also realized the value of intentional in-person touchpoints with colleagues to build culture and drive c


6 months ago @ medium.com
Large Scale Hadoop Upgrade At Pinterest
Large Scale Hadoop Upgrade At Pinterest Large Scale Hadoop Upgrade At Pinterest

Yongjun Zhang | Software Engineer; William Tom | Software Engineer; Shaowen Wang | Software Engineer; Bhavin Pathak | Software Engineer; Batch Processing Platform TeamPinterest’s Batch Processing Platform, Monarch, consists of more than 30 Hadoop YARN clusters with 17k+ nodes built entirely on top of AWS EC2. At the beginning of 2021, Monarch was still on Hadoop 2.7.1, which was already five years old. Because of the increasing complexity in backporting upstream changes (features and bug fixes), we decided it was time to invest in a version upgrade. We settled on Hadoop 2.10.0, which was the latest release of Hadoop 2 at the time.This article shares our experience of upgrading Monarch to Ha


6 months, 1 week ago @ medium.com
Facebook
Last posted 2 weeks, 1 day ago
Scaling data ingestion for machine learning training at Meta
Scaling data ingestion for machine learning training at Meta

Many of Meta’s products, such as search, ads ranking and Marketplace, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastructure increases, we need to scale our data ingestion infrastructure accordingly to handle workloads more efficiently. GPUs, which are used for training infrastructure, tend to double [...]

Read More...

The post Scaling data ingestion for machine learning training at Meta appeared first on Engineering at Meta.

2 weeks, 1 day ago @ engineering.fb.com
Applying federated learning to protect data on mobile devices
Applying federated learning to protect data on mobile devices

What the research is: Federated learning with differential privacy (FL-DP) is one of the latest privacy-enhancing technologies being evaluated at Meta as we constantly work to enhance user privacy and further safeguard users’ data in the products we design, build, and maintain. FL-DP enhances privacy in two important ways: It allows machine learning (ML) models [...]

Read More...

The post Applying federated learning to protect data on mobile devices appeared first on Engineering at Meta.

3 months, 3 weeks ago @ engineering.fb.com
VESPA: Static profiling for binary optimization
VESPA: Static profiling for binary optimization

What the research is: Recent research has demonstrated that binary optimization is important for achieving peak performance for various applications. For instance, the state-of-the-art BOLT binary optimizer developed at Meta, which is part of the LLVM Compiler Project, significantly improves the performance of highly optimized binaries produced using compilers’ most aggressive optimizations, such as profile-guided [...]

Read More...

The post VESPA: Static profiling for binary optimization appeared first on Engineering at Meta.

6 months, 3 weeks ago @ engineering.fb.com
Uber Engineering
Last posted None
Spotify Engineering
Last posted 5 days, 10 hours ago
Protected: Lessons Learned from Algorithmic Impact Assessments in Practice
Protected: Lessons Learned from Algorithmic Impact Assessments in Practice

There is no excerpt because this is a protected post.

The post Protected: Lessons Learned from Algorithmic Impact Assessments in Practice appeared first on Spotify Engineering.

5 days, 10 hours ago @ engineering.atspotify.com
From Development to Real Users: How to Create a Web Performance Story
From Development to Real Users: How to Create a Web Performance Story

Some of the most common questions asked when it comes to work with performance are, How do you convince stakeholders that improving the performance of your project is actually worth the investment? How can you prove that the work is necessary to begin with? Or prove that you have shipped improvements? And what is the [...]

The post From Development to Real Users: How to Create a Web Performance Story appeared first on Spotify Engineering.

5 days, 14 hours ago @ engineering.atspotify.com
Scaling Translations at Spotify
Scaling Translations at Spotify Scaling Translations at Spotify

Last year, we added support for 36 new languages to our products in one go, for a total of 62 languages. This article describes how we delivered on such an immense localization effort at Spotify. We called the project Scaling Translations. The business case We believe that localization is key for engaging with our users [...]

The post Scaling Translations at Spotify appeared first on Spotify Engineering.

2 weeks, 5 days ago @ engineering.atspotify.com
How We Maintain Security Testing within the Software Development Life Cycle
How We Maintain Security Testing within the Software Development Life Cycle

TL;DR The software development life cycle (SDLC) has always been followed by functional testing to ensure software solutions have all the necessary features and functions. Because of the growing number of cyberattacks, software development stakeholders have been forced to implement security testing as the main track in SDLC to prevent vulnerabilities and flaws in applications [...]

The post How We Maintain Security Testing within the Software Development Life Cycle appeared first on Spotify Engineering.

1 month, 1 week ago @ engineering.atspotify.com
Software Visualization — Challenge, Accepted
Software Visualization — Challenge, Accepted

TL;DR Architectural diagrams are the bread and butter of software design and a foundational tool for communication and collaboration on software development. At Spotify, we have an incredibly complex network of thousands of interlinked software systems owned by hundreds of teams, so having a simple way to visualize these connections is essential. While capturing all [...]

The post Software Visualization — Challenge, Accepted appeared first on Spotify Engineering.

2 months, 1 week ago @ engineering.atspotify.com
SCRIBD
Last posted 2 months, 2 weeks ago
Data and AI Summit Wrap-up
Data and AI Summit Wrap-up Data and AI Summit Wrap-up

We brought a whole team to San Francisco to present and attend this year’s Data and

AI Summit, and it was a blast! I

would consider the event a success both in the attendance to the Scribd hosted

talks and the number of talks which discussed patterns we have adopted in our

own data and ML platform.

The three talks I wrote about

previously were well received and have

since been posted to YouTube along with hundreds of other talks. Christian Williams shared some of the

work he has done developing

kafka-delta-ingest in his talk: QP Hou, Scribd Emeritus, presented on

his foundational work to ensure correctness within delta-rs during his session: R Tyler Croy co-presented with Gavin

Edgley from 


2 months, 2 weeks ago @ tech.scribd.com
Accelerating Looker with Databricks SQL Serverless
Accelerating Looker with Databricks SQL Serverless

We recently migrated Looker to a Databricks SQL Serverless, improving our

infrastructure cost and reducing the footprint of infrastructure we need to

worry about! “Databricks SQL” which provides a single load balanced Endpoint

for executing Spark SQL queries across multiple Spark clusters behind the

scenes. “Serverless” is an evolution of that concept, rather than running a SQL

Endpoint in our AWS infrastructure, the entirety of execution happens on the

Databricks side. With a much simpler and faster interface, queries executed in

Looker now return results much faster to our users than ever before!

When we originally provisioned our “Databricks SQL” endpoints, we worked

together with our co


3 months, 1 week ago @ tech.scribd.com
Scribd is presenting at Data and AI Summit 2022
Scribd is presenting at Data and AI Summit 2022

We are very excited to be presenting and attending this year’s Data and AI

Summit which will be

hosted virtually and physically in San Francisco from June 27th-30th.

Throughout the course of 2021 we completed a number of really interesting

projects built around delta-rs and the

Databricks platform which we are thrilled to share with a broader audience.

In addition to the presentations listed below, a number of Scribd engineers who

are responsible for data and ML platform, machine learning systems, and more,

will be in attendance if you want to meet up and learn more about how Scribd

uses data and ML to change the way the world reads! Christian Williams will be sharing some of the

work he ha


5 months, 1 week ago @ tech.scribd.com
Atlassian Atlassian
Last posted 1 day, 6 hours ago
Working Together to Build Trust in our Cloud Ecosystem
Working Together to Build Trust in our Cloud Ecosystem Working Together to Build Trust in our Cloud Ecosystem

What does cloud trust have to do with apps?

Ultimately, cloud apps come with a shared responsibility.

Step 1: Make sure your app is meeting our cloud security requirementsIf you have a cloud app on the Atlassian Marketplace, you are expected to abide by our cloud app security requirements, which are updated regularly to ensure alignment with industry standards.

The latest updates to our cloud security requirements go into effect at the end of October, so please be sure that your cloud apps are meeting them.

If you're ready to go a step further in your trust journey, read up on the Cloud Fortified apps program, which provides apps with a badge on the Marketplace in exchange for additional in


1 day, 6 hours ago @ blog.developer.atlassian.com
Forge Roadmap Webinar Recap: Q3 2022
Forge Roadmap Webinar Recap: Q3 2022 Forge Roadmap Webinar Recap: Q3 2022

On August 23rd, we gathered together again for our quarterly webinar discussing our progress on the Forge Roadmap.

Since our webinar in Q1 2022, we've delivered 9 Forge Jira modules and have made progress on four more.

During Q1, we planned to release 7 new Forge modules in Q3.

Achieving Custom UI Bridge parityShipped In Progress Upcoming Custom UI Bridge N/A N/AOne big accomplishment this quarter was a popular ask from our community: reaching feature parity with Connect for the Custom UI bridge.

This JavaScript API for Custom UI apps enables interactions with the host product, APIs, and the user’s browser.

1 week, 1 day ago @ blog.developer.atlassian.com
AMA recap: Building and growing on Forge with Daniel Wester
AMA recap: Building and growing on Forge with Daniel Wester AMA recap: Building and growing on Forge with Daniel Wester

The highs of being an app developer[The best part of being an app developer is] the enjoyment of making other people's work lives easier.

As a Forge early adopter, Daniel had some advice to offer the community about Forge – both the current state and looking ahead to the long term roadmap.

The Atlassian developer community is made up of thousands of individuals who build and succeed together.

People make the Atlassian community what it is – and we're grateful for Daniel and all of the other folks who dedicate their time.

We hope you can join us for our next AMA coming soon on the developer community.

1 week, 1 day ago @ blog.developer.atlassian.com
Forge the Future of Teamwork — Codegeist 2022 is Live!
Forge the Future of Teamwork — Codegeist 2022 is Live! Forge the Future of Teamwork — Codegeist 2022 is Live!

Most Valuable Feedback Prizes $1,000 USD to 10 participants that provided actionable feedback about the Forge platform.

Build with Forge Forge, our next-gen app development platform, just celebrated one year of being generally available.

During last year's Codegeist, you submitted stellar feedback in the Forge Feedback category, which directly contributed to improvements in Forge.

In addition to prizes, Codegeist helps participants build their network, collaborate with peers and colleagues, find potential mentors, and get inspired.

Whether you're new to the Atlassian developer community or a seasoned expert, we encourage you to test your creativity and win BIG.

3 weeks, 6 days ago @ blog.developer.atlassian.com
Four Essential Technical Skills for First-time Atlassian App Builders
Four Essential Technical Skills for First-time Atlassian App Builders Four Essential Technical Skills for First-time Atlassian App Builders

Reading Time: 4 minutesAre you a developer looking to jump in and build your first Atlassian app?

Before you get started, let's learn a little bit about Forge, Atlassian's serverless app development platform.

Building a Forge app is a fantastic way to grow as a developer and support Atlassian app users with better ways of working.

However, the flexibility of the Forge platform means tools are available to bridge knowledge gaps.

There's a whole wealth of resources to keep you learning and building apps: extensive documentation, Atlassian support, and the Atlassian developer community.

1 month ago @ blog.developer.atlassian.com
Six ways we’re making Custom UI Bridge more powerful
Six ways we’re making Custom UI Bridge more powerful Six ways we’re making Custom UI Bridge more powerful

The Custom UI Bridge is a JavaScript API that allows you to seamlessly integrate those experiences with Atlassian products.

Over the past few months, we've made a number of enhancements to the Custom UI Bridge.

Permission to do moreFinally, we've added two new permissions to the Custom UI iframe: clipboard-write and display-capture .

For now, this new functionality is only available in Custom UI but we plan on making it available to UI Kit apps in the future.

The new and improved Custom UI Bridge is part of our ongoing initiative to bring your favourite Atlassian Connect features over to Forge.

1 month, 2 weeks ago @ blog.developer.atlassian.com
Announcing our 2022 Developer AMA Series
Announcing our 2022 Developer AMA Series Announcing our 2022 Developer AMA Series

Reading Time: 4 minutesThe Atlassian Developer Community is made up of people who have many talents: technical skills, business acumen, product specific knowledge, and much more.

Our 2022 Developer AMA series is an opportunity to ask three community members questions so you can learn from them and grow your own skills.

Now let's get to know our Developer AMA hosts!

During his time in the Atlassian Ecosystem, Daniel built apps for Jira, Bamboo, Bitbucket and Confluence using P2, Connect, and now Forge.

September 1-7, 2022 — The Atlassian CommunityMeet Tim Pettersen, Head of Developer Experience at AtlassianThroughout his career, Tim has interwoven platform engineering, developer advocacy, an


1 month, 3 weeks ago @ blog.developer.atlassian.com
Watch, Learn, Grow: 5 bonus talks from Developer Day 2022
Watch, Learn, Grow: 5 bonus talks from Developer Day 2022 Watch, Learn, Grow: 5 bonus talks from Developer Day 2022

You can still watch all the talks from Developer Day 2022 on our YouTube channel, as well as a host of new bonus talks that are continuously being added.

Here's a roundup of our five bonus talks, available now on our Developer Day 2022 YouTube channel.

TwoForge on Forge: How the Forge team builds Forge apps at AtlassianThe development experience on Forge matters to us.

We couldn't have done Developer Day 2022 without inspirational speakers on the day and bonus talk contributors like the folks above.

Head over to Developer Day 2022 videos on YouTube to get inspired and watch these five bonus talk videos – plus much more from the event.

2 months ago @ blog.developer.atlassian.com
How to Configure CI/CD for an Atlassian Forge App
How to Configure CI/CD for an Atlassian Forge App How to Configure CI/CD for an Atlassian Forge App

Change to the app subdirectory to see the app files: cd hello-world-app2/ Install your Forge appTo use your app, it must be installed onto an Atlassian site.

The forge install command then installs the deployed app onto an Atlassian site with the required API access.

Note, you must run the forge deploy command before forge install because an installation links your deployed app to an Atlassian site.

Navigate to the app’s top-level directory and deploy your app by running: forge deploy Install your app by running: forge install Select your Atlassian product using the arrow keys and press the enter key.

Because a Forge app is based on Node.js, the base Docker image for your pipeline configura


2 months, 1 week ago @ blog.developer.atlassian.com
Creating a Strong Product Demo for Customers: Tips from Nikki Zavadska at Jexo
Creating a Strong Product Demo for Customers: Tips from Nikki Zavadska at Jexo Creating a Strong Product Demo for Customers: Tips from Nikki Zavadska at Jexo

Each presenter has 5 minutes to share their app demo, and customers vote for the app demo they liked best and can connect for more information.

First things first, I wanted to thank the Atlassian team for organizing Appy Hours and supporting Atlassian developers.

How did you prepare for the demo you gave for Appy Hours, and what was the experience like?

I was pleasantly surprised by how many people joined Appy Hours and all the support we had during demos.

Sign up to demo your app at a future Appy Hours, and RSVP for our July Appy Hour event!

2 months, 1 week ago @ blog.developer.atlassian.com
Infrastructure
AWS
Last posted 8 hours ago
Redact sensitive data from streaming data in near-real time using Amazon Comprehend and Amazon Kinesis Data Firehose
Redact sensitive data from streaming data in near-real time using Amazon Comprehend and Amazon Kinesis Data Firehose

Near-real-time delivery of data and insights enable businesses to rapidly respond to their customers’ needs. Real-time data can come from a variety of sources, including social media, IoT devices, infrastructure monitoring, call center monitoring, and more. Due to the breadth and depth of data being ingested from multiple sources, businesses look for solutions to protect [
]

8 hours ago @ aws.amazon.com
Manage your Amazon QuickSight datasets more efficiently with the new user interface
Manage your Amazon QuickSight datasets more efficiently with the new user interface

Amazon QuickSight has launched a new user interface for dataset management. Previously, the dataset management experience was a popup dialog modal with limited space, and all functionality was displayed in this one small modal. The new dataset management experience replaces the existing popup dialog with a full-page experience, providing a clearer breakdown of a dataset’s [
]

10 hours ago @ aws.amazon.com
Automate data archival for Amazon Redshift time series tables
Automate data archival for Amazon Redshift time series tables

Amazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that makes it simple and cost-effective to analyze all of your data using standard SQL. Tens of thousands of customers today rely on Amazon Redshift to analyze exabytes of data and run complex analytical queries, making it the most widely used cloud [
]

11 hours ago @ aws.amazon.com
Reduce cost and development time with Amazon SageMaker Pipelines local mode
Reduce cost and development time with Amazon SageMaker Pipelines local mode

Creating robust and reusable machine learning (ML) pipelines can be a complex and time-consuming process. Developers usually test their processing and training scripts locally, but the pipelines themselves are typically tested in the cloud. Creating and running a full pipeline during experimentation adds unwanted overhead and cost to the development lifecycle. In this post, we [
]

11 hours ago @ aws.amazon.com
Build Oracle Enterprise Manager with a repository in an Amazon RDS Custom for Oracle database
Build Oracle Enterprise Manager with a repository in an Amazon RDS Custom for Oracle database

As you migrate Oracle workloads to AWS, you may want to implement Oracle Enterprise Manager (OEM) Cloud Control, Oracle’s management platform, which provides a single pane of glass for managing Oracle environments. In this post, we provide the architecture and process to implement OEM 13.5 with high availability (HA) with an Amazon Relational Database Service [
]

14 hours ago @ aws.amazon.com
Stream data with Amazon DocumentDB and Amazon MSK using a Kafka connector
Stream data with Amazon DocumentDB and Amazon MSK using a Kafka connector

A common trend in modern application development and data processing is the use of Apache Kafka as a standard delivery mechanism for your data pipeline and fan-out approach. Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed, highly available, and secure service that makes it simple for developers and DevOps managers to [
]

1 day, 10 hours ago @ aws.amazon.com
Design a data mesh with event streaming for real-time recommendations on AWS
Design a data mesh with event streaming for real-time recommendations on AWS

This blog post was co-authored with Federico Piccinini. The data landscape has been changing in recent years: there is a proliferation of entities producing and consuming large quantities of data within companies, and for most of them defining a proper data strategy has become of fundamental importance. A modern data strategy gives you a comprehensive [
]

1 day, 11 hours ago @ aws.amazon.com
Create high-quality data for ML models with Amazon SageMaker Ground Truth
Create high-quality data for ML models with Amazon SageMaker Ground Truth

Machine learning (ML) has improved business across industries in recent years—from the recommendation system on your Prime Video account, to document summarization and efficient search with Alexa’s voice assistance. However, the question remains of how to incorporate this technology into your business. Unlike traditional rule-based methods, ML automatically infers patterns from data so as to [
]

1 day, 11 hours ago @ aws.amazon.com
Build, Test and Deploy ETL solutions using AWS Glue and AWS CDK based CI/CD pipelines
Build, Test and Deploy ETL solutions using AWS Glue and AWS CDK based CI/CD pipelines

AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning (ML), and application development. It’s serverless, so there’s no infrastructure to set up or manage. This post provides a step-by-step guide to build a continuous integration and continuous delivery (CI/CD) pipeline using AWS [
]

1 day, 11 hours ago @ aws.amazon.com
Automate your time series forecasting in Snowflake using Amazon Forecast
Automate your time series forecasting in Snowflake using Amazon Forecast

This post is a joint collaboration with Andries Engelbrecht and James Sun of Snowflake, Inc. The cloud computing revolution has enabled businesses to capture and retain corporate and organizational data without capacity planning or data retention constraints. Now, with diverse and vast reserves of longitudinal data, companies are increasingly able to find novel and impactful [
]

1 day, 11 hours ago @ aws.amazon.com
Achieve four times higher ML inference throughput at three times lower cost per inference with Amazon EC2 G5 instances for NLP and CV PyTorch models
Achieve four times higher ML inference throughput at three times lower cost per inference with Amazon EC2 G5 instances for NLP and CV PyTorch models

Amazon Elastic Compute Cloud (Amazon EC2) G5 instances are the first and only instances in the cloud to feature NVIDIA A10G Tensor Core GPUs, which you can use for a wide range of graphics-intensive and machine learning (ML) use cases. With G5 instances, ML customers get high performance and a cost-efficient infrastructure to train and [
]

1 day, 11 hours ago @ aws.amazon.com
The five most visited Amazon DynamoDB blog posts of 2022
The five most visited Amazon DynamoDB blog posts of 2022

From January through September of 2022, Amazon Web Services (AWS) customers visited the following five Amazon DynamoDB blog posts more than all others. Use this list (starting with most visited) to see what other customers have been reading and decide what to learn next. Amazon DynamoDB can now import Amazon S3 data into a new [
]

1 day, 16 hours ago @ aws.amazon.com
Celebrate over 20 years of AI/ML at Innovation Day
Celebrate over 20 years of AI/ML at Innovation Day

Be our guest as we celebrate 20 years of AI/ML innovation on October 25, 2022, 9:00 AM – 10:30 AM PT. The first 1,500 people to register will receive $50 of AWS credits. Register here. Over the past 20 years, Amazon has delivered many world firsts for artificial intelligence (AI) and machine learning (ML). ML [
]

4 days, 5 hours ago @ aws.amazon.com
AWS Panorama now supports NVIDIA JetPack SDK 4.6.2
AWS Panorama now supports NVIDIA JetPack SDK 4.6.2

AWS Panorama is a collection of machine learning (ML) devices and a software development kit (SDK) that brings computer vision to on-premises internet protocol (IP) cameras. AWS Panorama device options include the AWS Panorama Appliance and the Lenovo ThinkEdge SE70, powered by AWS Panorama. These device options provide you choices in price and performance, depending [
]

4 days, 8 hours ago @ aws.amazon.com
Modeling a scalable fantasy football database with Amazon DynamoDB
Modeling a scalable fantasy football database with Amazon DynamoDB

Today’s online games generate more data than ever and have request rates that reach millions per second. For these data-intensive games, it’s important for developers to select a database that delivers consistent low latency at any scale and has throughput elasticity to accommodate spikes in traffic without costly overprovisioning during low activity periods. This is [
]

4 days, 10 hours ago @ aws.amazon.com
AWS
Last posted 8 hours ago
Build flexible and scalable distributed training architectures using Kubeflow on AWS and Amazon SageMaker
Build flexible and scalable distributed training architectures using Kubeflow on AWS and Amazon SageMaker

In this post, we demonstrate how Kubeflow on AWS (an AWS-specific distribution of Kubeflow) used with AWS Deep Learning Containers and Amazon Elastic File System (Amazon EFS) simplifies collaboration and provides flexibility in training deep learning models at scale on both Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon SageMaker utilizing a hybrid architecture approach. [
]

4 days, 10 hours ago @ aws.amazon.com
Bundesliga Match Fact Pressure Handling: Evaluating players’ performances in high-pressure situations on AWS
Bundesliga Match Fact Pressure Handling: Evaluating players’ performances in high-pressure situations on AWS

Pressing or pressure in football is a process in which a team seeks to apply stress to the opponent player who possesses the ball. A team applies pressure to limit the time an opposition player has left to make a decision, reduce passing options, and ultimately attempt to turn over ball possession. Although nearly all [
]

4 days, 12 hours ago @ aws.amazon.com
Bundesliga Match Fact Win Probability: Quantifying the effect of in-game events on winning chances using machine learning on AWS
Bundesliga Match Fact Win Probability: Quantifying the effect of in-game events on winning chances using machine learning on AWS

Ten years from now, the technological fitness of clubs will be a key contributor towards their success. Today we’re already witnessing the potential of technology to revolutionize the understanding of football. xGoals quantifies and allows comparison of goal scoring potential of any shooting situation, while xThreat and EPV models predict the value of any in-game [
]

4 days, 12 hours ago @ aws.amazon.com
Unified data preparation, model training, and deployment with Amazon SageMaker Data Wrangler and Amazon SageMaker Autopilot – Part 2
Unified data preparation, model training, and deployment with Amazon SageMaker Data Wrangler and Amazon SageMaker Autopilot – Part 2

Depending on the quality and complexity of data, data scientists spend between 45–80% of their time on data preparation tasks. This implies that data preparation and cleansing take valuable time away from real data science work. After a machine learning (ML) model is trained with prepared data and readied for deployment, data scientists must often [
]

4 days, 13 hours ago @ aws.amazon.com
How Sophos trains a powerful, lightweight PDF malware detector at ultra scale with Amazon SageMaker
How Sophos trains a powerful, lightweight PDF malware detector at ultra scale with Amazon SageMaker

This post is co-authored by Salma Taoufiq and Harini Kannan from Sophos. As a leader in next-generation cybersecurity, Sophos strives to protect more than 500,000 organizations and millions of customers across over 150 countries against evolving threats. Powered by threat intelligence, machine learning (ML), and artificial intelligence from Sophos X-Ops, Sophos delivers a broad and [
]

5 days, 10 hours ago @ aws.amazon.com
Build a high-performance, transactional data lake using open-source Delta Lake on Amazon EMR
Build a high-performance, transactional data lake using open-source Delta Lake on Amazon EMR

Data lakes on Amazon Simple Storage Service (Amazon S3) have become the default repository for all enterprise data and serve as a common choice for a large number of users querying from a variety of analytics and machine learning (ML) tools. Oftentimes you want to ingest data continuously into the data lake from multiple sources [
]

5 days, 10 hours ago @ aws.amazon.com
Understanding statistics in PostgreSQL
Understanding statistics in PostgreSQL

PostgreSQL has become the preferred open-source relational database for many enterprise developers and startups, and powers leading business and mobile applications. AWS provides two managed PostgreSQL options: Amazon Relational Database Service (Amazon RDS) for PostgreSQL and Amazon Aurora PostgreSQL-Compatible Edition. Database statistics play a key role in improving the performance of the database. The query [
]

5 days, 13 hours ago @ aws.amazon.com
Using CloudFormation events to build custom workflows for post provisioning management
Using CloudFormation events to build custom workflows for post provisioning management

Over one million active customers manage application resources with AWS CloudFormation every week. CloudFormation is a service that helps you model, provision, and manage your cloud resources by treating Infrastructure as Code (IaC). It can simplify infrastructure management, quickly replicate your environment to multiple AWS regions with a single turn-key solution, and let you easily [
]

6 days ago @ aws.amazon.com
Build an AI-powered virtual agent for Genesys Cloud using QnABot and Amazon Lex
Build an AI-powered virtual agent for Genesys Cloud using QnABot and Amazon Lex

The rise of artificial intelligence technologies enables organizations to adopt and improve self-service capabilities in contact center operations to create a more proactive, timely, and effective customer experience. Voice bots, or conversational interactive voice response systems (IVR), use natural language processing (NLP) to understand customers’ questions and provide relevant answers. Businesses can automate responses to [
]

6 days, 12 hours ago @ aws.amazon.com
Ensure availability of your data using cross-cluster replication with Amazon OpenSearch Service
Ensure availability of your data using cross-cluster replication with Amazon OpenSearch Service

Amazon OpenSearch Service is a fully managed service that you can use to deploy and operate OpenSearch and legacy Elasticsearch clusters, cost-effectively, at scale in the AWS Cloud. The service makes it easy for you to perform interactive log analytics, real-time application monitoring, website search, and more by offering the latest versions of OpenSearch, suppor300t [
]

6 days, 12 hours ago @ aws.amazon.com
Set up enterprise-level cost allocation for ML environments and workloads using resource tagging in Amazon SageMaker
Set up enterprise-level cost allocation for ML environments and workloads using resource tagging in Amazon SageMaker

As businesses and IT leaders look to accelerate the adoption of machine learning (ML), there is a growing need to understand spend and cost allocation for your ML environment to meet enterprise requirements. Without proper cost management and governance, your ML spend may lead to surprises in your monthly AWS bill. Amazon SageMaker is a [
]

6 days, 12 hours ago @ aws.amazon.com
Automated testing of Amazon Neptune data access with Apache TinkerPop Gremlin
Automated testing of Amazon Neptune data access with Apache TinkerPop Gremlin

Amazon Neptune, a fully managed graph database, is purpose built to work with highly connected data such as relationships between customers and products, or between pieces of equipment within a complex industrial plant. Neptune is designed to support highly concurrent online transaction processing (OLTP) over graph data models. Neptune supports both property graphs, which you [
]

6 days, 14 hours ago @ aws.amazon.com
Build high availability for Amazon RDS Custom for Oracle using read replicas
Build high availability for Amazon RDS Custom for Oracle using read replicas

A high availability solution for the database stack is an important aspect to consider when migrating or deploying Oracle databases in the AWS Cloud to help ensure that the architecture can meet the service level agreement (SLA) of the application. Unavailability of a critical database can lead to application outage, interruption to business operations, inaccessibility [
]

6 days, 14 hours ago @ aws.amazon.com
Index your Dropbox content using the Dropbox connector for Amazon Kendra
Index your Dropbox content using the Dropbox connector for Amazon Kendra

Amazon Kendra is a highly accurate and simple-to-use intelligent search service powered by machine learning (ML). Amazon Kendra offers a suite of data source connectors to simplify the process of ingesting and indexing your content, wherever it resides. Valuable data in organizations is stored in both structured and unstructured repositories. An enterprise search solution should [
]

1 week ago @ aws.amazon.com
Provision and manage ML environments with Amazon SageMaker Canvas using AWS CDK and AWS Service Catalog
Provision and manage ML environments with Amazon SageMaker Canvas using AWS CDK and AWS Service Catalog

The proliferation of machine learning (ML) across a wide range of use cases is becoming prevalent in every industry. However, this outpaces the increase in the number of ML practitioners who have traditionally been responsible for implementing these technical solutions to realize business outcomes. In today’s enterprise, there is a need for machine learning to [
]

1 week ago @ aws.amazon.com
Azure
Last posted 19 hours ago
Azure Firewall Basic now in preview
Azure Firewall Basic now in preview

Azure Firewall Basic is a new SKU of Azure Firewall designed to meet the needs of SMBs by providing enterprise-grade protection of their cloud environment at an affordable price point. It is a cloud-native, highly available, stateful firewall-as-a-service offering that enables customers to centrally govern and log all of their traffic flows with essential capabilities at scale.

19 hours ago @ azure.microsoft.com
Microsoft and INT deploy IVAAP for OSDU Data Platform on Microsoft Energy Data Services
Microsoft and INT deploy IVAAP for OSDU Data Platform on Microsoft Energy Data Services

With Microsoft Energy Data Services, energy companies can leverage new cloud-based advanced data visualization capabilities for geoscientists provided by INT and Microsoft Energy Data Services.

20 hours ago @ azure.microsoft.com
Advancing anomaly detection with AIOps—introducing AiDice
Advancing anomaly detection with AIOps—introducing AiDice

We introduce AiDice, a novel anomaly detection algorithm developed jointly by Microsoft Research and Microsoft Azure that identifies anomalies in large-scale, multi-dimensional time series data. AiDice captures incidents quickly and provides engineers with important context that helps them diagnose issues more effectively, providing the best experience possible for end customers.

1 day, 19 hours ago @ azure.microsoft.com
Strengthen your security with Policy Analytics for Azure Firewall
Strengthen your security with Policy Analytics for Azure Firewall

We are excited to share that Policy Analytics for Azure Firewall is now in preview. Policy Analytics provides you with critical insights and analytics to optimize your Azure Firewall rules and strengthen your security posture.

5 days, 19 hours ago @ azure.microsoft.com
Ensure zone resilient outbound connectivity with NAT gateway
Ensure zone resilient outbound connectivity with NAT gateway

When customers need to connect outbound to the internet from their Azure infrastructures, Network Address Translation (NAT) gateway is the best way. NAT gateway is a zonal resource that is configured to subnets from the same virtual network, which means that it can be deployed to individual zones to allow outbound connectivity.

5 days, 20 hours ago @ azure.microsoft.com
Cost Management updates—September 2022
Cost Management updates—September 2022

New in September are support for budgets in the Azure mobile app, cost savings insights in the cost analysis preview, and new licensing benefits for partners. Add 7 new GA updates and 3 doc updates and you have another great month.

6 days, 18 hours ago @ azure.microsoft.com
RoQC and Microsoft simplify cloud migration with Microsoft Energy Data Services
RoQC and Microsoft simplify cloud migration with Microsoft Energy Data Services

The vast amount of data in energy companies slows down their digital transformation. Together with RoQC solutions, Microsoft Energy Data Services will accelerate your journey in democratizing access to data by providing an easy to deploy managed service fully supported by Microsoft.

6 days, 20 hours ago @ azure.microsoft.com
New Azure for Operators solution accelerator offers a fast path to network insights
New Azure for Operators solution accelerator offers a fast path to network insights

Azure for Operators is introducing a network analytics solution accelerator program, providing a standardized approach to data acquisition and visualization that aids operators on their journey toward complete end-to-end AI Operations (AIOps).

1 week ago @ azure.microsoft.com
EPAM and Microsoft partner on data governance solutions with Microsoft Energy Data Services
EPAM and Microsoft partner on data governance solutions with Microsoft Energy Data Services

Microsoft Energy Data Services is a data platform fully supported by Microsoft, that enables efficient data management, standardization, liberation, and consumption in energy exploration. The solution is a hyperscale data ecosystem that leverages the capabilities of the OSDU Data Platform, Microsoft's secure and trustworthy cloud services with our partners’ extensive domain expertise.

1 week, 1 day ago @ azure.microsoft.com
Future-ready IoT implementations on Microsoft Azure
Future-ready IoT implementations on Microsoft Azure

Gartner has positioned Microsoft as a Leader in the Gartner Magic Quadrant for Industrial IoT Platforms for the second year in a row. And Frost and Sullivan named the platform the Global IoT platform of the year. In 2022, Microsoft continues to double down on Azure services to provide enterprises with business-critical IoT solutions.

1 week, 5 days ago @ azure.microsoft.com
Cegal and Microsoft break down data silos and offer open collaboration with Microsoft Energy Data Services
Cegal and Microsoft break down data silos and offer open collaboration with Microsoft Energy Data Services

The vast amount of applications and data in energy companies across isolated environments is exposing inefficiencies in collaboration. Together with Cegal Cetegra, Microsoft Energy Data Services will accelerate your journey in seamless access all data and applications you need for your day-to-day work by providing an easy-to-deploy managed service fully supported by Microsoft.

1 week, 5 days ago @ azure.microsoft.com
Azure Payment HSM achieves PCI PIN certification offering customers secure digital payments solutions in the cloud
Azure Payment HSM achieves PCI PIN certification offering customers secure digital payments solutions in the cloud

Today we’re announcing Azure Payment HSM has achieved Payment Card Industry Personal Identification Number (PCI PIN) making Azure the first hyperscale cloud service provider to obtain this certification.

1 week, 6 days ago @ azure.microsoft.com
Wipro and Microsoft partner on services and accelerators for the new Microsoft Energy Data Services
Wipro and Microsoft partner on services and accelerators for the new Microsoft Energy Data Services

With Microsoft Energy Data Services, Wipro is offering to adopt an open architecture that provides accelerated access to new technologies through an open, modular cloud agnostic design.

2 weeks ago @ azure.microsoft.com
Microsoft shares what's next in machine learning at NVIDIA GTC
Microsoft shares what's next in machine learning at NVIDIA GTC

Finding scalable solutions for today’s global challenges requires forward-thinking, transformative tools. As environmental, economic, and public health concerns mount, Microsoft Azure is addressing these challenges head on with high-performance computing (HPC), AI, and machine learning.

2 weeks, 6 days ago @ azure.microsoft.com
New Azure Space products enable digital resiliency and empower the industry
New Azure Space products enable digital resiliency and empower the industry

Since the launch of Azure Space two years ago, we’ve announced partnerships, products, and tools that have focused on how we can bring together the power of the cloud with the possibilities of space. Today, we are introducing the next wave of product advancements for this mission and announcing specific ways in which we are democratizing space and empowering our partners.

2 weeks, 6 days ago @ azure.microsoft.com