All Things Engineering
Engineering & Developer Experience from the best in class. Culture, Architecture, DevOps and stuff at scale.
🏢 %company% Engineering
AirBnb Engineering
Last posted 1 week ago
Learning To Rank Diversely
Learning To Rank Diversely Learning To Rank Diversely

by Malay Haldar, Liwei He & Moose AbdoolAirbnb connects millions of guests and Hosts everyday. Most of these connections are forged through search, the results of which are determined by a neural network–based ranking algorithm. While this neural network is adept at selecting individual listings for guests, we recently improved the neural network to better select the overall collection of listings that make up a search result. In this post, we dive deeper into this recent breakthrough that enhances the diversity of listings in search results.How Does Ranking Work?The ranking neural network finds the best listings to surface for a given query by comparing two listings at a time and predictin…

1 week ago @ medium.com
Making Airbnb’s Android app more accessible
Making Airbnb’s Android app more accessible Making Airbnb’s Android app more accessible

By: Julia Fu, Peter ElliottAt Airbnb, we have been consciously designing and building products to be equally usable by all users. Making our mobile apps and websites more accessible not only aligns with our company’s mission of creating a world where people can belong anywhere, but also supports the civil rights of people with disabilities and complies with the law.In this article, we highlight some of the efforts we have made to make the app more accessible, for example, labeling UI elements, grouping related content, supporting large font scale, providing heading and page names. The Airbnb app is one of the most popular travel apps with millions of users and supports many features. Making…

3 weeks, 5 days ago @ medium.com
When a Picture Is Worth More Than Words
When a Picture Is Worth More Than Words When a Picture Is Worth More Than Words

How Airbnb uses visual attributes to enhance the Guest and Host experienceBy Yuanpei Cao, Bill Ulammandakh, Hao Wang, and Tony HwangIntroductionOn Airbnb, our hosts share unique listings all over the world. There are hundreds of millions of accompanying listing photos on Airbnb. Listing photos contain crucial information about style and design aesthetics that are difficult to convey in words or a fixed list of amenities. Accordingly, multiple teams at Airbnb are now leveraging computer vision to extract and incorporate intangibles from our rich visual data to help guests easily find listings that suit their preferences.In previous blog posts titled WIDeText: A Multimodal Deep Learning Frame…

2 months ago @ medium.com
Motion Engineering at Scale
Motion Engineering at Scale Motion Engineering at Scale

How Airbnb is applying declarative design patterns to rapidly build fluid transition animationsBy: Cal StephensMotion is a key part of what makes a digital experience both easy and delightful to use. Fluid transitions between states and screens are key for helping the user preserve context as they navigate throughout a feature. Quick flourishes of animation make an app come alive, and help give it a distinct personality.At Airbnb we launch hundreds of features and experiments that have been developed by engineers across many teams. When building at this scale, it’s critical to consider efficiency and maintainability throughout our tech stack–and motion is no exception. Adding animations to …

2 months ago @ medium.com
Announcing Lottie 4.0 for iOS
Announcing Lottie 4.0 for iOS Announcing Lottie 4.0 for iOS

A new rendering engine with significant performance improvements powered by Core AnimationBy: Cal StephensLottie is Airbnb’s cross-platform, open source library for rendering vector motion graphics. We use Lottie extensively at Airbnb, and it also powers animations in thousands of other apps throughout the industry.Example Lottie animations included in Airbnb’s iOS appToday we’re releasing Lottie 4.0 for iOS. This major new release brings significant performance improvements to all Lottie animations, with a brand new rendering engine powered by Core Animation.Using Lottie at scale for many years, we’ve learned a lot about its performance characteristics in real-world use cases. We found tha…

2 months ago @ medium.com
How AI Text Generation Models Are Reshaping Customer Support at Airbnb
How AI Text Generation Models Are Reshaping Customer Support at Airbnb How AI Text Generation Models Are Reshaping Customer Support at Airbnb

Leveraging text generation models to build more effective, scalable customer support products.Gavin Li, Mia Zhao and Zhenyu ZhaoOne of the fastest-growing areas in modern Artificial Intelligence (AI) is AI text generation models. As the name suggests, these models generate natural language. Previously, most industrial natural language processing (NLP) models were classifiers, or what might be called discriminative models in machine learning (ML) literature. However, in recent years, generative models based on large-scale language models are rapidly gaining traction and fundamentally changing how ML problems are formulated. Generative models can now obtain some domain knowledge through large…

2 months, 2 weeks ago @ medium.com
Building Airbnb Categories with ML and Human-in-the-Loop
Building Airbnb Categories with ML and Human-in-the-Loop Building Airbnb Categories with ML and Human-in-the-Loop

Airbnb Categories Blog Series — Part IBy: Mihajlo Grbovic, Ying Xiao, Pratiksha Kadam, Aaron Yin, Pei Xiong, Dillon Davis, Aditya Mukherji, Kedar Bellare, Haowei Zhang, Shukun Yang, Chen Qian, Sebastien Dubois, Nate Ney, James Furnary, Mark Giangreco, Nate Rosenthal, Cole Baker, Bill Ulammandakh, Sid Reddy, Egor PakhomovFigure 1. Browsing listings by categories: Castles, Desert, Design, Beach & Countryside25 Years of Online Travel SearchOnline travel search hasn’t changed much in the last 25 years. The traveler enters her destination, dates, and the number of guests into a search interface, which dutifully returns a list of options that best meet the criteria. Eventually, Airbnb and other t…

2 months, 2 weeks ago @ medium.com
Mussel — Airbnb’s Key-Value Store for Derived Data
Mussel — Airbnb’s Key-Value Store for Derived Data Mussel — Airbnb’s Key-Value Store for Derived Data

Mussel — Airbnb’s Key-Value Store for Derived DataHow Airbnb built a persistent, high availability and low latency key-value storage engine for accessing derived data from offline and streaming events.By: Chandramouli Rangarajan, Shouyan Guo, Yuxi JinIntroductionWithin Airbnb, many online services need access to derived data, which is data computed with large scale data processing engines like Spark or streaming events like Kafka and stored offline. These services require a high quality derived data storage system, with strong reliability, availability, scalability, and latency guarantees for serving online traffic. For example, the user profiler service stores and accesses real-time and hi…

3 months, 4 weeks ago @ medium.com
Beyond A/B test : Speeding up Airbnb Search Ranking Experimentation through Interleaving
Beyond A/B test : Speeding up Airbnb Search Ranking Experimentation through Interleaving Beyond A/B test : Speeding up Airbnb Search Ranking Experimentation through Interleaving

Beyond A/B Test : Speeding up Airbnb Search Ranking Experimentation through InterleavingIntroduction of Airbnb interleaving experimentation framework, usage and approaches to address challenges in our unique businessQing Zhang, Michelle Du, Reid Andersen, Liwei HeIntroductionWhen a user searches for a place to stay on Airbnb, we aim to show them the best results possible. Airbnb’s relevance team actively works on improving search ranking experience and helps users to find and book listings that match their preference. A/B test is our approach for online assessment. Our business metrics are conversion-focused, and the frequency of guest travel transactions is lower than on other e-commerce p…

4 months ago @ medium.com
Upgrading Data Warehouse Infrastructure at Airbnb
Upgrading Data Warehouse Infrastructure at Airbnb Upgrading Data Warehouse Infrastructure at Airbnb

This blog aims to introduce Airbnb’s experience upgrading Data Warehouse infrastructure to Spark and Iceberg.By: Ronnie Zhu, Edgar Rodriguez, Jason Xu, Gustavo Torres, Kerim Oktay, Xu ZhangIntroductionIn this blog, we will introduce our motivations for upgrading our Data Warehouse Infrastructure to Spark 3 and Iceberg. We will briefly describe the current state of Airbnb data warehouse infrastructure and the challenges. We will then share our learnings from upgrading one critical production workload: event data ingestion. Finally, we will share the results and the lessons learned.ContextAirbnb’s Data Warehouse (DW) storage was previously migrated from legacy HDFS clusters to S3 to provide b…

4 months, 1 week ago @ medium.com
Netflix Engineering Netflix Engineering
Last posted 1 week ago
Discovering Creative Insights in Promotional Artwork
Discovering Creative Insights in Promotional Artwork Discovering Creative Insights in Promotional Artwork

By Grace Tang, Aneesh Vartakavi, Julija Bagdonaite, Cristina Segalin, and Vi IyengarWhen members are shown a title on Netflix, the displayed artwork, trailers, and synopses are personalized. That means members see the assets that are most likely to help them make an informed choice. These assets are a critical source of information for the member to make a decision to watch, or not watch, a title. The stories on Netflix are multidimensional and there are many ways that a single story could appeal to different members. We want to show members the images, trailers, and synopses that are most helpful to them for making a watch decision.In a previous blog post we explained how our artwork perso…

1 week ago @ netflixtechblog.com
Scalable Annotation Service — Marken
Scalable Annotation Service — Marken Scalable Annotation Service — Marken

Scalable Annotation Service — Markenby Varun Sekhri, Meenakshi JindalIntroductionAt Netflix, we have hundreds of micro services each with its own data models or entities. For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. All of these services at a later point want to annotate their objects or entities. Our team, Asset Management Platform, decided to create a generic service called Marken which allows any microservice at Netflix to annotate their entity.AnnotationsSometimes people describe annotations as tags but that is a limited definition. In Marken, an annotation is a piece of metadata which can be attached to an object f…

1 week, 4 days ago @ netflixtechblog.com
Ready-to-go sample data pipelines with Dataflow
Ready-to-go sample data pipelines with Dataflow Ready-to-go sample data pipelines with Dataflow

by Jasmine Omeke, Obi-Ike Nwoke, Olek GorajekIntroThis post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix.You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow. That article was a deep dive into one of the more technical aspects of Dataflow and didn’t properly introduce this tool in the first place. This time we’ll try to give justice to the intro and then we will focus on one of the very first features Dataflow came with. That feature is called sample workflows, but before we start in let’s have a quick look at Dataflow in general.D…

2 months ago @ netflixtechblog.com
For your eyes only: improving Netflix video quality with neural networks
For your eyes only: improving Netflix video quality with neural networks For your eyes only: improving Netflix video quality with neural networks

by Christos G. Bampis, Li-Heng Chen and Zhi LiWhen you are binge-watching the latest season of Stranger Things or Ozark, we strive to deliver the best possible video quality to your eyes. To do so, we continuously push the boundaries of streaming video quality and leverage the best video technologies. For example, we invest in next-generation, royalty-free codecs and sophisticated video encoding optimizations. Recently, we added another powerful tool to our arsenal: neural networks for video downscaling. In this tech blog, we describe how we improved Netflix video quality with neural networks, the challenges we faced and what lies ahead.How can neural networks fit into Netflix video encodin…

2 months, 3 weeks ago @ netflixtechblog.com
Match Cutting at Netflix: Finding Cuts with Smooth Visual Transitions
Match Cutting at Netflix: Finding Cuts with Smooth Visual Transitions Match Cutting at Netflix: Finding Cuts with Smooth Visual Transitions

Match Cutting: Finding Cuts with Smooth Visual Transitions Using Machine LearningBy Boris Chen, Kelli Griggs, Amir Ziai, Yuchen Xie, Becky Tucker, Vi Iyengar, Ritwik Kumar, Keila Fong, Nagendra Kamath, Elliot Chow, Robert Mayer, Eugene Lok, Aly Parmelee, Sarah BlankCreating Media with Machine Learning episode 1IntroductionAt Netflix, part of what we do is build tools to help our creatives make exciting videos to share with the world. Today, we’d like to share some of the work we’ve been doing on match cuts.https://medium.com/media/f0a78d7f38bd14f60761a93c160f92bd/hrefIn film, a match cut is a transition between two shots that uses similar visual framing, composition, or action to fluidly br…

2 months, 3 weeks ago @ netflixtechblog.com
Helping VFX studios pave a path to the cloud
Helping VFX studios pave a path to the cloud Helping VFX studios pave a path to the cloud

By: Peter Cioni (Netflix), Alex Schworer (Netflix), Mac Moore (Conductor Tech.), Rachel Kelley (AWS), Ranjit Raju (AWS)Rendering is core to the VFX processVFX studios around the world create amazing imagery for Netflix productions. Nearly every show that is produced today includes digital visual effects, from the creatures in Stranger Things, to recreating historic London in Bridgerton.Netflix production teams work with a global roster of VFX studios (both large and small) and their artists to create this amazing imagery. But it’s not easy: to pull this off, VFX studios need to build and operate serious technical infrastructure (compute, storage, networking, and software licensing), otherwi…

2 months, 3 weeks ago @ netflixtechblog.com
New Series: Creating Media with Machine Learning
New Series: Creating Media with Machine Learning New Series: Creating Media with Machine Learning

By Vi Iyengar, Keila Fong, Hossein Taghavi, Andy Yao, Kelli Griggs, Boris Chen, Cristina Segalin, Apurva Kansara, Grace Tang, Billur Engin, Amir Ziai, James Ray, Jonathan Solorzano-HamiltonWelcome to the first post in our multi-part series on how Netflix is developing and using machine learning (ML) to help creators make better media — from TV shows to trailers to movies to promotional art and so much more.Media is at the heart of Netflix. It’s our medium for delivering a range of emotions and experiences to our members. Through each engagement, media is how we bring our members continued joy.This blog series will take you behind the scenes, showing you how we use the power of machine learn…

2 months, 3 weeks ago @ netflixtechblog.com
Machine Learning for Fraud Detection in Streaming Services
Machine Learning for Fraud Detection in Streaming Services Machine Learning for Fraud Detection in Streaming Services

By Soheil Esmaeilzadeh, Negin Salajegheh, Amir Ziai, Jeff BooteIntroductionStreaming services serve content to millions of users all over the world. These services allow users to stream or download content across a broad category of devices including mobile phones, laptops, and televisions. However, some restrictions are in place, such as the number of active devices, the number of streams, and the number of downloaded titles. Many users across many platforms make for a uniquely large attack surface that includes content fraud, account fraud, and abuse of terms of service. Detection of fraud and abuse at scale and in real-time is highly challenging.Data analysis and machine learning techniq…

2 months, 3 weeks ago @ netflixtechblog.com
Seeing through hardware counters: a journey to threefold performance increase
Seeing through hardware counters: a journey to threefold performance increase Seeing through hardware counters: a journey to threefold performance increase

By Vadim Filanovsky and Harshad SaneIn one of our previous blogposts, A Microscope on Microservices we outlined three broad domains of observability (or “levels of magnification,” as we referred to them) — Fleet-wide, Microservice and Instance. We described the tools and techniques we use to gain insight within each domain. There is, however, a class of problems that requires an even stronger level of magnification going deeper down the stack to introspect CPU microarchitecture. In this blogpost we describe one such problem and the tools we used to solve it.The problemIt started off as a routine migration. At Netflix, we periodically reevaluate our workloads to optimize utilization of avail…

2 months, 4 weeks ago @ netflixtechblog.com
Consistent caching mechanism in Titus Gateway
Consistent caching mechanism in Titus Gateway Consistent caching mechanism in Titus Gateway

by Tomasz Bak and Fabio KungIntroductionTitus is the Netflix cloud container runtime that runs and manages containers at scale. In the time since it was first presented as an advanced Mesos framework, Titus has transparently evolved from being built on top of Mesos to Kubernetes, handling an ever-increasing volume of containers. As the number of Titus users increased over the years, the load and pressure on the system increased substantially. The original assumptions and architectural choices were no longer viable. This blog post presents how our current iteration of Titus deals with high API call volumes by scaling out horizontally.We introduce a caching mechanism in the API gateway layer,…

3 months ago @ netflixtechblog.com
Pinterest Engineering
Last posted 1 week, 4 days ago
Employee-facing Mutual TLS
Employee-facing Mutual TLS Employee-facing Mutual TLS

Armen Tashjian | Security Engineer, Corporate SecurityThis blog article is the second part of our recently released blog: Enforcing Device AuthN & Compliance at Pinterest.IntroAs part of our device authentication and compliance initiative, Pinterest has implemented employee-facing mutual TLS with a custom identity provider in a way that results in a positive user experience.You may have heard of, or experienced first hand, some unpleasant behavior while attempting to authenticate with a certificate within a browser or application. Even the Wikipedia page for mutual TLS mentions that mTLS is a “..less user-friendly experience, [and] it’s rarely used in end-user applications…”.At Pinterest, w…

1 week, 4 days ago @ medium.com
Enforcing Device AuthN & Compliance at Pinterest
Enforcing Device AuthN & Compliance at Pinterest Enforcing Device AuthN & Compliance at Pinterest

Armen Tashjian | Security Engineer, Corporate SecurityIntroPinterest has enforced the use of managed and compliant devices in our Okta authentication flow, using a passwordless implementation, so that access to our tools always requires a healthy Pinterest device.Following the phishing-based attacks against our peers in the tech industry, Pinterest decided to take a two pronged approach to defend against similar attacks. We decided to:Require a managed and healthy Pinterest device be used to access all Pinterest resources, even when in the possession of valid credentialsRequire FIDO2 credentials for user authenticationIn this post, we’ll be focusing on how we required the use of Pinterest m…

1 week, 6 days ago @ medium.com
Build an end to end JSON logging system for clients apps
Build an end to end JSON logging system for clients apps Build an end to end JSON logging system for clients apps

Liang Ma | Software Engineer, Core Eng; Wei Zhu | Software Engineer, ObservabilityIn early 2020, during a critical iOS out of memory incident (we have a blogpost for that), we realized that we didn’t have much visibility of how the app is running or a good system to look up for monitoring and troubleshooting.State of loggingAt that time, on the client side, there were a few ways for logging in their daily work:Context logging: built for logging and reporting impressions or anything related to business, thus a time critical and first-class endpoint. Developers need to explicitly define keys that would otherwise be rejected by the endpoint. Some companies call it “analytics logging.”Misc: log…

3 weeks, 6 days ago @ medium.com
Improving the Player on Android
Improving the Player on Android Improving the Player on Android

Grey Skold | (former Android Video Engineer) ; Lin Wang | Android Performance Engineer; Sheng Liu | Android Performance EngineerPinterest Android App offers a rare experience with a mix of images and videos on a two-column grid. In order to maintain a performant video experience on Android devices, we focused on:Warming upConfigurationsPooling playersWarming UpIn order to reduce the startup latency, we establish a video network connection by sending a dummy HTTP HEAD request during the early application startup time. The same connection can be used to play future videos. This is done even before any video urls are returned from our server.The same strategy also applies to UI rendering. We f…

2 months ago @ medium.com
Flexible Daily Budgeting at Pinterest
Flexible Daily Budgeting at Pinterest Flexible Daily Budgeting at Pinterest

Kelvin Jiang, Software Engineer, Ads Intelligence | Keshava Subramanya, Engineering Manager, Ads Intelligence | Perrye Ogunwole, Product Manager, Ads IntelligenceIntroductionThe Ads Intelligence team at Pinterest builds products that help advertisers maximize the value they get out of their ad campaigns. As part of that initiative, we have recently launched Flexible Daily Budgets (FDB) to US advertisers in open beta.Figure 1: Flexible daily budgets option in the campaign create flowFDB is an automated ads product that benefits advertisers by distributing the advertising budget for each campaign across multiple days in an automated manner. The goals of this product are to:Enable advertisers …

2 months, 3 weeks ago @ medium.com
How Pinterest Leverages Realtime User Actions in Recommendation to Boost Homefeed Engagement Volume
How Pinterest Leverages Realtime User Actions in Recommendation to Boost Homefeed Engagement Volume How Pinterest Leverages Realtime User Actions in Recommendation to Boost Homefeed Engagement Volume

Xue Xia, Software Engineer, Homefeed Ranking; Neng Gu, Software Engineer, Content & User Understanding; Dhruvil Deven Badani, Engineering Manager, Homefeed Ranking; Andrew Zhai, Software Engineer, Advanced Technologies GroupImage from https://wallpapercave.com/neural-networks-wallpapers#google_vignetteIn this blog post, we will demonstrate how we improved Pinterest Homefeed engagement volume from a machine learning model design perspective — by leveraging realtime user action features in Homefeed recommender system.BackgroundThe Homepage of Pinterest is the one of most important surfaces for pinners to discover inspirational ideas and contributes to a large fraction of overall user engageme…

3 months ago @ medium.com
Watch your Manifest
Watch your Manifest Watch your Manifest

Lin Wang | Android Performance EngineerDesigned by AJ Oxendine | Software EngineerIt’s a well-known fact for Android developers that an app’s manifest (AndroidManifest.xml) holds crucial application declarations. It is rarely monitored after being set up because we assume it hardly ever changes. At Pinterest, however, we have been actively monitoring the manifest after realizing it does change every so often.While building an app, Gradle downloads all the dependent libraries to compile and link them with the app. These dependent libraries each have their own mini manifest. During the build process, Android Gradle Plugin (AGP) merges them with the app’s main manifest to form the final manife…

3 months, 1 week ago @ medium.com
Query Rewards: Building a Recommendation Feedback Loop During Query Selection
Query Rewards: Building a Recommendation Feedback Loop During Query Selection Query Rewards: Building a Recommendation Feedback Loop During Query Selection

Bella Huang | Software Engineer, Home Candidate Generation; Raymond Hsu | Engineer Manager, Home Candidate Generation; Dylan Wang | Engineer Manager, Home RelevanceIn Homefeed, ~30% of recommended pins come from pin to pin-based retrieval. This means that during the retrieval stage, we use a batch of query pins to call our retrieval system to generate pin recommendations. We typically use a user’s previously engaged pins, and a user may have hundreds (or thousands!) of engaged pins, so a key problem for us is: how do we select the right query pins from the user’s profile?User Profiling with PinnerSAGE OverviewAt Pinterest, we use PinnerSAGE as the main source of a user’s pin profile. Pinner…

3 months, 1 week ago @ medium.com
My Summer as a Software Engineering Intern at Pinterest Toronto!
My Summer as a Software Engineering Intern at Pinterest Toronto! My Summer as a Software Engineering Intern at Pinterest Toronto!

Khubi Shah | (former) Software Engineer Intern, Shopping Content MiningThis summer, I had the incredible opportunity to intern at the one and only Pinterest from the new engineering hub in Toronto!I am a final year undergraduate student from the University of Waterloo, majoring in Computer Science with an AI specialization. Growing up, Pinterest was always my go-to social media platform, as it inspired me with new ideas for food, fashion, design, or anything creative! As a result, when I was given the opportunity to join as a Software Engineering Intern this summer in Toronto, I was beyond thrilled to be a part of the company mission: to bring everyone the inspiration to create a life they …

3 months, 3 weeks ago @ medium.com
Online Data Migration from HBase to TiDB with Zero Downtime
Online Data Migration from HBase to TiDB with Zero Downtime Online Data Migration from HBase to TiDB with Zero Downtime

Ankita Girish Wagh | Senior Software Engineer, Storage and CachingIntroduction and MotivationAt Pinterest, HBase is one of the most critical storage backends, powering many online storage services like Zen (graph database), UMS (wide column datastore), and Ixia (near real time secondary indexing service). The HBase Ecosystem, though having various advantages like strong consistency at row level in high volume requests, flexible schema, low latency access to data, Hadoop integration, etc. cannot serve the needs of our clients for the next 3–5 years. This is due to high operational cost, excessive complexity, and missing functionalities like secondary indexes, support for transactions, etc.Af…

5 months, 3 weeks ago @ medium.com
Facebook
Last posted 3 months, 1 week ago
Improving Instagram notification management with machine learning and causal inference
Improving Instagram notification management with machine learning and causal inference

We’re sharing how Meta is applying statistics and machine learning (ML) to improve notification personalization and management on Instagram – particularly on daily digest push notifications. By using causal inference and ML to identify highly active users who are likely to see more content organically, we have been able to reduce the number of notifications [...]

Read More...

The post Improving Instagram notification management with machine learning and causal inference appeared first on Engineering at Meta.

3 months, 1 week ago @ engineering.fb.com
Scaling data ingestion for machine learning training at Meta
Scaling data ingestion for machine learning training at Meta

Many of Meta’s products, such as search, ads ranking and Marketplace, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastructure increases, we need to scale our data ingestion infrastructure accordingly to handle workloads more efficiently. GPUs, which are used for training infrastructure, tend to double [...]

Read More...

The post Scaling data ingestion for machine learning training at Meta appeared first on Engineering at Meta.

4 months, 2 weeks ago @ engineering.fb.com
Applying federated learning to protect data on mobile devices
Applying federated learning to protect data on mobile devices

What the research is: Federated learning with differential privacy (FL-DP) is one of the latest privacy-enhancing technologies being evaluated at Meta as we constantly work to enhance user privacy and further safeguard users’ data in the products we design, build, and maintain. FL-DP enhances privacy in two important ways: It allows machine learning (ML) models [...]

Read More...

The post Applying federated learning to protect data on mobile devices appeared first on Engineering at Meta.

7 months, 3 weeks ago @ engineering.fb.com
Uber Engineering
Last posted None
Spotify Engineering Spotify Engineering
Last posted 5 days, 6 hours ago
Unleashing ML Innovation at Spotify with Ray
Unleashing ML Innovation at Spotify with Ray

Introduction As the field of machine learning (ML) continues to evolve and its impact on society and various aspects of our lives grows, it is becoming increasingly important for practitioners and innovators to consider a broader range of perspectives when building ML models and applications. This desire is driving the need for a more flexible [...]

The post Unleashing ML Innovation at Spotify with Ray appeared first on Spotify Engineering.

5 days, 6 hours ago @ engineering.atspotify.com
Agile Coaching (AC) at Spotify: Shining a Light on the AC Career Framework
Agile Coaching (AC) at Spotify: Shining a Light on the AC Career Framework

At Spotify, change is a constant, and our agile coaches are here to help teams and leaders thrive as we evolve. With agile principles famously part of our DNA, Spotify doesn’t need “transformation coaches,” as they are often known in the industry, but coaches who transform as organizational needs change. In the early 2010s, our [...]

The post Agile Coaching (AC) at Spotify: Shining a Light on the AC Career Framework appeared first on Spotify Engineering.

2 weeks, 5 days ago @ engineering.atspotify.com
What’s a “Listening Personality”?
What’s a “Listening Personality”?

We did a couple of new things in Wrapped this year, and one of these is a thing called Your Listening Personality. You got one. There are 16 of them, in total: It’s not a lottery or a horoscope, though. As the four-letter codes at the bottom of each one hint, each Listening Personality is [...]

The post What’s a “Listening Personality”? appeared first on Spotify Engineering.

1 month ago @ engineering.atspotify.com
Shipping Spotify’s Culture: 5 Plugins (and 4 Principles) for Supercharging Developer Experience at Scale
Shipping Spotify’s Culture: 5 Plugins (and 4 Principles) for Supercharging Developer Experience at Scale

ICYMI Spotify officially began selling enterprise software today: the Spotify Plugins for Backstage bundle subscription. Some of these commercial plugins began life as internal tools, built by Spotify developers, for Spotify developers. Other plugins in the bundle are brand new to the Backstage platform. But they all embody our ways of working — our secret [...]

The post Shipping Spotify’s Culture: 5 Plugins (and 4 Principles) for Supercharging Developer Experience at Scale appeared first on Spotify Engineering.

1 month, 3 weeks ago @ engineering.atspotify.com
Strategies and Tools for Performing Migrations on Platform
Strategies and Tools for Performing Migrations on Platform

Needing to deliver faster and more reliably while managing a growing number of contributors and a more complex codebase seems like the fate of every hyper-growth tech company. For platform teams, the challenge is not any different. How can we quickly roll out and increase the adoption of new technologies safely with a growing codebase [...]

The post Strategies and Tools for Performing Migrations on Platform appeared first on Spotify Engineering.

2 months, 3 weeks ago @ engineering.atspotify.com
SCRIBD
Last posted 6 months, 2 weeks ago
Data and AI Summit Wrap-up
Data and AI Summit Wrap-up Data and AI Summit Wrap-up

We brought a whole team to San Francisco to present and attend this year’s Data and

AI Summit, and it was a blast! I

would consider the event a success both in the attendance to the Scribd hosted

talks and the number of talks which discussed patterns we have adopted in our

own data and ML platform.

The three talks I wrote about

previously were well received and have

since been posted to YouTube along with hundreds of other talks. Christian Williams shared some of the

work he has done developing

kafka-delta-ingest in his talk: QP Hou, Scribd Emeritus, presented on

his foundational work to ensure correctness within delta-rs during his session: R Tyler Croy co-presented with Gavin

Edgley from …

6 months, 2 weeks ago @ tech.scribd.com
Accelerating Looker with Databricks SQL Serverless
Accelerating Looker with Databricks SQL Serverless

We recently migrated Looker to a Databricks SQL Serverless, improving our

infrastructure cost and reducing the footprint of infrastructure we need to

worry about! “Databricks SQL” which provides a single load balanced Endpoint

for executing Spark SQL queries across multiple Spark clusters behind the

scenes. “Serverless” is an evolution of that concept, rather than running a SQL

Endpoint in our AWS infrastructure, the entirety of execution happens on the

Databricks side. With a much simpler and faster interface, queries executed in

Looker now return results much faster to our users than ever before!

When we originally provisioned our “Databricks SQL” endpoints, we worked

together with our co…

7 months, 1 week ago @ tech.scribd.com
Scribd is presenting at Data and AI Summit 2022
Scribd is presenting at Data and AI Summit 2022

We are very excited to be presenting and attending this year’s Data and AI

Summit which will be

hosted virtually and physically in San Francisco from June 27th-30th.

Throughout the course of 2021 we completed a number of really interesting

projects built around delta-rs and the

Databricks platform which we are thrilled to share with a broader audience.

In addition to the presentations listed below, a number of Scribd engineers who

are responsible for data and ML platform, machine learning systems, and more,

will be in attendance if you want to meet up and learn more about how Scribd

uses data and ML to change the way the world reads! Christian Williams will be sharing some of the

work he ha…

9 months, 2 weeks ago @ tech.scribd.com
Atlassian Atlassian
Last posted 5 days, 5 hours ago
Reminder: New server app sales are ending on February 15th
Reminder: New server app sales are ending on February 15th Reminder: New server app sales are ending on February 15th

Of course, this transition will require changes on the Atlassian Marketplace, including the end of server app sales.

To ensure all server app contracts are complete by the server end of support deadline, new server app sales (including free apps) will end on February 15, 2023.

This change applies to apps for Jira Software server, Jira Service Management server, Confluence server, and Bitbucket server.

New server app sales will endOn February 15 you will see that the "Try it free" and "Buy it now" buttons will no longer be visible on paid server app listings, and the "Get it now" button will no longer be visible on free server app listings.

In the meantime, if you have questions about the en…

5 days, 5 hours ago @ blog.developer.atlassian.com
January 2023: Marketplace Partner Program Tier Membership Update
January 2023: Marketplace Partner Program Tier Membership Update January 2023: Marketplace Partner Program Tier Membership Update

– Marketplace Partners are investing in cloud and have Data Center Approved versions of their high-traction server apps.

Security – Marketplace Partners are security-minded and improving their security practices by participating in our Marketplace Security Programs.

Cloud Security Participant badge and Cloud Fortified badge were introduced for participating cloud apps on Marketplace so customers can make informed decisions about the security of the cloud apps.

As we usher yet another exciting year for Atlassian Marketplace, let us welcome the latest partners who have achieved their medal tiers in the Marketplace Partner Program following our 6-month program level review.

*Bolded are Marketp…

2 weeks ago @ blog.developer.atlassian.com
New improvements to app reviews on the Atlassian Marketplace
New improvements to app reviews on the Atlassian Marketplace New improvements to app reviews on the Atlassian Marketplace

Reading Time: 3 minutesApp reviews are an important part of the Atlassian Marketplace.

While we know most Marketplace Partners understand and respect this need for true customer feedback, we also know we can do more to make Marketplace app reviews more trustworthy.

Today, we're announcing the latest step toward more trustworthy app reviews: reviewer verification.

Ultimately, we expect this review step to reduce the incidence of fake reviews on the Marketplace.

What we'll do nextIn addition to adding a verification step on the Marketplace, we are working on strengthening our policies around what constitutes fair behavior on the Marketplace.

2 weeks, 6 days ago @ blog.developer.atlassian.com
Partner preview: a new Privacy & Security Tab is coming to the Marketplace
Partner preview: a new Privacy & Security Tab is coming to the Marketplace Partner preview: a new Privacy & Security Tab is coming to the Marketplace

Reading Time: 4 minutesThis quarter, we are planning to add a new Privacy & Security tab to Marketplace cloud app listings.

Why are we adding a Privacy & Security tab?

Assessing app security and privacy is a key step in the Marketplace app procurement process for many Atlassian cloud customers and migrating customers.

How can you fill out the Privacy & Security tab for your apps?

The Privacy & Security tab will surface information that will be useful to a majority of customers.

1 month ago @ blog.developer.atlassian.com
Forge Roadmap Webinar Recap: Q4 2022
Forge Roadmap Webinar Recap: Q4 2022 Forge Roadmap Webinar Recap: Q4 2022

To give developers a better sense of the services running behind the scenes and unpack the implications for reliability, Colin did a deep dive into the Forge platform architecture.

Forge roadmap: What's coming nextMoving on to new features, Forge product managers previewed a number of new capabilities coming to Forge, including: Client-side UI kit, multi-user app ownership, and changes to app data access.

Developers will be able to build UI kit apps and take advantage of native mobile rendering support for selected extension points and UI kit components.

If you're interested in trying Forge Mobile, look for our early access announcement in early 2023.

We've begun a new EAP for a version of …

1 month, 1 week ago @ blog.developer.atlassian.com
Looking Back at 2022: The Atlassian Developer Community Year in Review
Looking Back at 2022: The Atlassian Developer Community Year in Review Looking Back at 2022: The Atlassian Developer Community Year in Review

For the Atlassian Developer Community, 2022 was a year of growth and adapting to continual change.

We brought our builders together through community events and contests, enhanced the capabilities of our platforms, and worked towards a streamlined Atlassian developer experience.

Let's take a tour of a few key events, product updates, and announcements in our Developer Community.

Excited to get involved in the Developer Community?

Launched a consolidated Developer Support PortalSpeaking of consolidation, 2022 also brought some important changes to our developer support with our new Developer Support Portal.

1 month, 2 weeks ago @ blog.developer.atlassian.com
From fun project to profitable side hustle: Taking your app to the Marketplace
From fun project to profitable side hustle: Taking your app to the Marketplace From fun project to profitable side hustle: Taking your app to the Marketplace

You'll need to be sure you can dedicate the time and resources to testing your app, building a brand, and potentially customer service too.

Once you list your first Paid via Atlassian app, you'll also get access to even more customer insights to help you succeed on the Marketplace.

Finally, all apps in the Atlassian Marketplace need to meet the legal and compliance requirements laid out in the Marketplace Partner Agreement, plus any additional legal requirements depending on where and how you do business.

Getting an app to the Marketplace for the first time can be a lot of new things to think about.

You can also check out our Developer Showcase to see some Marketplace app success stories fr…

1 month, 3 weeks ago @ blog.developer.atlassian.com
Let’s Git down to business at App Week ’23!
Let’s Git down to business at App Week ’23! Let’s Git down to business at App Week ’23!

Applications have just opened for App Week ’23, taking place in Berlin from February 6-February 10, 2023.

App Week ’23 is the place to try new developer platform features, learn best practices for enterprise-grade Cloud apps, and discuss all things Open DevOps.

Applicants should have an app in mind that you want to build or improve on during your time at App Week.

App Week '23 is focused on building apps that will support customers’ journey to Cloud.

In order to maintain the quality of the event, we can only host 100 attendees at App Week.

2 months ago @ blog.developer.atlassian.com
Start building app editions with new build instructions for Connect apps
Start building app editions with new build instructions for Connect apps Start building app editions with new build instructions for Connect apps

App editions will make it possible to list multiple editions of a cloud app under a single Marketplace listing.

Start building editions for your Connect apps in the backgroundThis week we are releasing technical guides in the Partner Portal to help you get started on the next stage of your app editions journey: the build phase.

If you have a product strategy in place and you're ready to start building editions for your Connect app(s), you can use these new instructions to start building out your editions in the background.

This initial set of build instructions is for apps built on Connect only.

What are Marketplace app editions?

2 months ago @ blog.developer.atlassian.com
A new way to stay up to date with the Developer Changelog
A new way to stay up to date with the Developer Changelog A new way to stay up to date with the Developer Changelog

Reading Time: 2 minutesAs the number of changelogs across different Atlassian cloud products continues to grow, we're excited to announce that we have added a landing page, the Atlassian Developer Changelog, that consolidates all other Atlassian Developer changelogs and displays changes across multiple products.

Using your input to shape the new experienceWe hear you: keeping track of multiple changelogs across the multiple documentation sets on Atlassian Developer is not optimal.

The Atlassian Developer Changelog will unify API change announcements from all existing changelogs, and is flexible enough to display only what you want to see, no more or less.

Starting today, the new page is acc…

2 months, 1 week ago @ blog.developer.atlassian.com
Infrastructure
AWS
Last posted 2 hours ago
Deep dive into the AWS ProServe Hadoop Migration Delivery Kit TCO tool
Deep dive into the AWS ProServe Hadoop Migration Delivery Kit TCO tool

In the post Introducing the AWS ProServe Hadoop Migration Delivery Kit TCO tool, we introduced the AWS ProServe Hadoop Migration Delivery Kit (HMDK) TCO tool and the benefits of migrating on-premises Hadoop workloads to Amazon EMR. In this post, we dive deep into the tool, walking through all steps from log ingestion, transformation, visualization, and […]

2 hours ago @ aws.amazon.com
Introducing the AWS ProServe Hadoop Migration Delivery Kit TCO tool
Introducing the AWS ProServe Hadoop Migration Delivery Kit TCO tool

When migrating Hadoop workloads to Amazon EMR, it’s often difficult to identify the optimal cluster configuration without analyzing existing workloads by hand. To solve this, we’re introducing the Hadoop migration assessment Total Cost of Ownership (TCO) tool. You now have a Hadoop migration assessment TCO tool within the AWS ProServe Hadoop Migration Delivery Kit (HMDK). […]

2 hours ago @ aws.amazon.com
Improve observability across Amazon MWAA tasks
Improve observability across Amazon MWAA tasks

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that makes it simple to set up and operate end-to-end data pipelines in the cloud at scale. A data pipeline is a set of tasks and processes used to automate the movement and transformation of data between different systems.­ […]

3 hours ago @ aws.amazon.com
Image classification model selection using Amazon SageMaker JumpStart
Image classification model selection using Amazon SageMaker JumpStart

Researchers continue to develop new model architectures for common machine learning (ML) tasks. One such task is image classification, where images are accepted as input and the model attempts to classify the image as a whole with object label outputs. With many models available today that perform this image classification task, an ML practitioner may […]

3 hours ago @ aws.amazon.com
Introducing MongoDB Atlas metadata collection with AWS Glue crawlers
Introducing MongoDB Atlas metadata collection with AWS Glue crawlers

For data lake customers who need to discover petabytes of data, AWS Glue crawlers are a popular way to discover and catalog data in the background. This allows users to search and find relevant data from multiple data sources. Many customers also have data in managed operational databases such as MongoDB Atlas and need to […]

4 hours ago @ aws.amazon.com
Enable Kerberos authentication with Amazon RDS Custom for Oracle – Part 2
Enable Kerberos authentication with Amazon RDS Custom for Oracle – Part 2

In the first post of this series, Enable Kerberos authentication with Amazon RDS Custom for Oracle – Part 1, we showed you how to implement Kerberos authentication with Amazon RDS Custom for Oracle to centralize the storage and management of credentials across multiple databases and improve your organization’s overall security profile. In this post, we […]

3 days, 3 hours ago @ aws.amazon.com
Migrate your Azure Cosmos DB SQL API to Amazon DocumentDB (with MongoDB compatibility) using native tools and automation
Migrate your Azure Cosmos DB SQL API to Amazon DocumentDB (with MongoDB compatibility) using native tools and automation

While migrating workloads from the Azure Cloud to the AWS Cloud, organizations explore optimal, managed database services to replace their Cosmos DB databases. As NoSQL databases become more ubiquitous, especially those that support the Apache 2.0 open-source MongoDB APIs, our customers often choose Amazon DocumentDB because it’s a scalable, highly durable, and fully managed database […]

3 days, 3 hours ago @ aws.amazon.com
Predict football punt and kickoff return yards with fat-tailed distribution using GluonTS
Predict football punt and kickoff return yards with fat-tailed distribution using GluonTS

Today, the NFL is continuing their journey to increase the number of statistics provided by the Next Gen Stats Platform to all 32 teams and fans alike. With advanced analytics derived from machine learning (ML), the NFL is creating new ways to quantify football, and to provide fans with the tools needed to increase their […]

3 days, 23 hours ago @ aws.amazon.com
Analyze and visualize multi-camera events using Amazon SageMaker Studio Lab
Analyze and visualize multi-camera events using Amazon SageMaker Studio Lab

The National Football League (NFL) is one of the most popular sports leagues in the United States and is the most valuable sports league in the world. The NFL, BioCore, and AWS are committed to advancing human understanding around the diagnosis, prevention, and treatment of sports-related injuries to make the game of football safer. More […]

3 days, 23 hours ago @ aws.amazon.com
Analyze Amazon S3 storage costs using AWS Cost and Usage Reports, Amazon S3 Inventory, and Amazon Athena
Analyze Amazon S3 storage costs using AWS Cost and Usage Reports, Amazon S3 Inventory, and Amazon Athena

Since its launch in 2006, Amazon Simple Storage Service (Amazon S3) has experienced major growth, supporting multiple use cases such as hosting websites, creating data lakes, serving as object storage for consumer applications, storing logs, and archiving data. As the application portfolio grows, customers tend to store data from multiple application and different business functions […]

4 days ago @ aws.amazon.com
Accelerate your multi-region strategy with Amazon DynamoDB: Part 1
Accelerate your multi-region strategy with Amazon DynamoDB: Part 1

When you decide to run applications across multiple geographic regions, there are many questions you need to answer. First, what are the business drivers? Resiliency? It might turn out that your application requirements for resiliency, such as availability and disaster recovery, can be satisfied within a single AWS Region. If you do need a multi-region […]

4 days, 3 hours ago @ aws.amazon.com
How Amazon Devices scaled and optimized real-time demand and supply forecasts using serverless analytics
How Amazon Devices scaled and optimized real-time demand and supply forecasts using serverless analytics

Every day, Amazon devices process and analyze billions of transactions from global shipping, inventory, capacity, supply, sales, marketing, producers, and customer service teams. This data is used in procuring devices’ inventory to meet Amazon customers’ demands. With data volumes exhibiting a double-digit percentage growth rate year on year and the COVID pandemic disrupting global logistics […]

5 days ago @ aws.amazon.com
How to decide between Amazon Rekognition image and video API for video moderation
How to decide between Amazon Rekognition image and video API for video moderation

Almost 80% of today’s web content is user-generated, creating a deluge of content that organizations struggle to analyze with human-only processes. The availability of consumer information helps them make decisions, from buying a new pair of jeans to securing home loans. In a recent survey, 79% of consumers stated they rely on user videos, comments, […]

5 days, 1 hour ago @ aws.amazon.com
Amazon EMR launches support for Amazon EC2 C7g (Graviton3) instances to improve cost performance for Spark workloads by 7–13%
Amazon EMR launches support for Amazon EC2 C7g (Graviton3) instances to improve cost performance for Spark workloads by 7–13%

Amazon EMR provides a managed service to easily run analytics applications using open-source frameworks such as Apache Spark, Hive, Presto, Trino, HBase, and Flink. The Amazon EMR runtime for Spark and Presto includes optimizations that provide over twice the performance improvements compared to open-source Apache Spark and Presto. With Amazon EMR release 6.7, you can […]

5 days, 3 hours ago @ aws.amazon.com
Scaling distributed training with AWS Trainium and Amazon EKS
Scaling distributed training with AWS Trainium and Amazon EKS

Recent developments in deep learning have led to increasingly large models such as GPT-3, BLOOM, and OPT, some of which are already in excess of 100 billion parameters. Although larger models tend to be more powerful, training such models requires significant computational resources. Even with the use of advanced distributed training libraries like FSDP and […]

5 days, 3 hours ago @ aws.amazon.com
AWS
Last posted 2 hours ago
Improve observability by using Amazon RDS Custom for SQL Server with Telegraf and Amazon Grafana
Improve observability by using Amazon RDS Custom for SQL Server with Telegraf and Amazon Grafana

You can use open source monitoring solutions like Telegraf, InfluxDB, and Grafana to monitor your applications and databases running on-premises or on Amazon Elastic Compute Cloud (Amazon EC2). In this post, we explain how you can leverage these tools on Amazon Relational Database Service (Amazon RDS) Custom for SQL Server. Use of open source software […]

5 days, 6 hours ago @ aws.amazon.com
AWS Lake Formation 2022 year in review
AWS Lake Formation 2022 year in review

Data governance is the collection of policies, processes, and systems that organizations use to ensure the quality and appropriate handling of their data throughout its lifecycle for the purpose of generating business value. Data governance is increasingly top-of-mind for customers as they recognize data as one of their most important assets. Effective data governance enables […]

5 days, 21 hours ago @ aws.amazon.com
Visualize multivariate data using a radar chart in Amazon QuickSight
Visualize multivariate data using a radar chart in Amazon QuickSight

At AWS re:Invent 2022, we announced the general availability of two new Amazon QuickSight visuals: small multiples and text boxes. We are excited to add another new visual to QuickSight: radar charts. With radar charts, you can compare two or more items across multiple variables in QuickSight. In this post, we explore radar charts, its […]

6 days ago @ aws.amazon.com
Migrate your indexes to Amazon OpenSearch Serverless with Logstash
Migrate your indexes to Amazon OpenSearch Serverless with Logstash

We recently announced the general availability of Amazon OpenSearch Serverless , a new option for Amazon OpenSearch Service that makes it easy run large-scale search and analytics workloads without having to configure, manage, or scale OpenSearch clusters. With OpenSearch Serverless, you get the same interactive millisecond response times as OpenSearch Service with the simplicity of […]

6 days ago @ aws.amazon.com
Serverless logging with Amazon OpenSearch Service and Amazon Kinesis Data Firehose
Serverless logging with Amazon OpenSearch Service and Amazon Kinesis Data Firehose

In this post, you will learn how you can use Amazon Kinesis Data Firehose to build a log ingestion pipeline to send VPC flow logs to Amazon OpenSearch Serverless. First, you create the OpenSearch Serverless collection you use to store VPC flow logs, then you create a Kinesis Data Firehose delivery pipeline that forwards the […]

6 days ago @ aws.amazon.com
Create more partitions and retain data for longer in your MSK Serverless clusters
Create more partitions and retain data for longer in your MSK Serverless clusters

In April 2022, Amazon Managed Streaming for Apache Kafka (Amazon MSK) launched an exciting new capability, Amazon MSK Serverless. Amazon MSK is a fully managed service for Apache Kafka that makes it easier for developers to build and run highly available, secure, and scalable applications based on Apache Kafka. With MSK Serverless, developers can run […]

6 days, 22 hours ago @ aws.amazon.com
Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 3: Summary and best practices)
Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 3: Summary and best practices)

In Part 1 of this series, you learned about Amazon DynamoDB data loading strategies and the behavior of DynamoDB during short runs. In Part 2, you learned about query performance and the adaptive behavior of DynamoDB during sustained activity. In this third and final post, we review what you’ve learned, plus offer a few additional […]

1 week ago @ aws.amazon.com
Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 2: Querying)
Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 2: Querying)

In the Part 1 of this series, you learned about Amazon DynamoDB data loading strategies and the behavior of DynamoDB during short runs. In this post, you learn about query performance and the adaptive behavior of DynamoDB during sustained activity. Querying To drive traffic at arbitrarily large rates and simulate real-world behavior, we need a […]

1 week ago @ aws.amazon.com
Run Apache Spark workloads 3.5 times faster with Amazon EMR 6.9
Run Apache Spark workloads 3.5 times faster with Amazon EMR 6.9

In this post, we analyze the results from our benchmark tests running a TPC-DS application on open-source Apache Spark and then on Amazon EMR 6.9, which comes with an optimized Spark runtime that is compatible with open-source Spark. We walk through a detailed cost analysis and finally provide step-by-step instructions to run the benchmark. With Amazon EMR 6.9.0, you can now run your Apache Spark 3.x applications faster and at lower cost without requiring any changes to your applications. In our performance benchmark tests, derived from TPC-DS performance tests at 3 TB scale, we found the EMR runtime for Apache Spark 3.3.0 provides a 3.5 times (using total runtime) performance improvement o…

1 week ago @ aws.amazon.com
Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 1: Loading)
Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 1: Loading)

The general rule with Amazon DynamoDB is to choose a high cardinality partition key. But why; and what happens if you don’t? Inspired by a customer use case, we dive deep into this question and explore the performance of loading and querying DynamoDB using different partition key designs and table settings. After each experiment, we […]

1 week ago @ aws.amazon.com
Handle UPSERT data operations using open-source Delta Lake and AWS Glue
Handle UPSERT data operations using open-source Delta Lake and AWS Glue

Many customers need an ACID transaction (atomic, consistent, isolated, durable) data lake that can log change data capture (CDC) from operational data sources. There is also demand for merging real-time data into batch data. Delta Lake framework provides these two capabilities. In this post, we discuss how to handle UPSERTs (updates and inserts) of the […]

1 week ago @ aws.amazon.com
Amazon SageMaker built-in LightGBM now offers distributed training using Dask
Amazon SageMaker built-in LightGBM now offers distributed training using Dask

Amazon SageMaker provides a suite of built-in algorithms, pre-trained models, and pre-built solution templates to help data scientists and machine learning (ML) practitioners get started on training and deploying ML models quickly. You can use these algorithms and models for both supervised and unsupervised learning. They can process various types of input data, including tabular, […]

1 week ago @ aws.amazon.com
Build a water consumption forecasting solution for a water utility agency using Amazon Forecast
Build a water consumption forecasting solution for a water utility agency using Amazon Forecast

Amazon Forecast is a fully managed service that uses machine learning (ML) to generate highly accurate forecasts, without requiring any prior ML experience. Forecast is applicable in a wide variety of use cases, including estimating supply and demand for inventory management, travel demand forecasting, workforce planning, and computing cloud infrastructure usage. You can use Forecast […]

1 week ago @ aws.amazon.com
Build a data lake with Apache Flink on Amazon EMR
Build a data lake with Apache Flink on Amazon EMR

To build a data-driven business, it is important to democratize enterprise data assets in a data catalog. With a unified data catalog, you can quickly search datasets and figure out data schema, data format, and location. The AWS Glue Data Catalog provides a uniform repository where disparate systems can store and find metadata to keep […]

1 week, 3 days ago @ aws.amazon.com
Advanced reporting and analytics for the Post Call Analytics (PCA) solution with Amazon QuickSight
Advanced reporting and analytics for the Post Call Analytics (PCA) solution with Amazon QuickSight

Organizations with contact centers benefit from advanced analytics on their call recordings to gain important product feedback, improve contact center efficiency, and identify coaching opportunities for their staff. The Post Call Analytics (PCA) solution uses AWS machine learning (ML) services like Amazon Transcribe and Amazon Comprehend to extract insights from contact center call audio recordings […]

1 week, 3 days ago @ aws.amazon.com
Azure
Last posted 4 days, 11 hours ago
Scale Azure Firewall SNAT ports with NAT Gateway for large workloads
Scale Azure Firewall SNAT ports with NAT Gateway for large workloads

When it comes to providing outbound connectivity to the internet from cloud architectures using Azure Firewall, look no further than NAT gateway.

4 days, 11 hours ago @ azure.microsoft.com
Microsoft named a Leader in the IDC MarketScape: Worldwide MLOps Platforms 2022 Vendor Assessment
Microsoft named a Leader in the IDC MarketScape: Worldwide MLOps Platforms 2022 Vendor Assessment

We’re excited to share that Microsoft has been recognized as a Leader in the IDC MarketScape Worldwide Machine Learning Operations (MLOps) Platforms 2022 Vendor Assessment.

5 days, 5 hours ago @ azure.microsoft.com
Microsoft Azure Load Testing is now generally available
Microsoft Azure Load Testing is now generally available

We are announcing the general availability of Azure Load Testing, a fully managed load-testing service.

5 days, 10 hours ago @ azure.microsoft.com
Azure Native NGINXaas makes traffic management secure and simple—now generally available
Azure Native NGINXaas makes traffic management secure and simple—now generally available

Continuing Microsoft Azure’s commitment to empower our ISV partners and customers to adopt and modernize their application of choice and run in the cloud, we are excited to announce general availability (GA) of the NGINXaaS offering on Azure.

5 days, 11 hours ago @ azure.microsoft.com
Azure high-performance computing powers energy industry innovation
Azure high-performance computing powers energy industry innovation

Azure High-Performance Computing provides a platform for energy industry innovation at scale.

5 days, 13 hours ago @ azure.microsoft.com
Microsoft Cost Management updates—January 2023
Microsoft Cost Management updates—January 2023

January is always exciting, and this year is no different with the general availability of EA account management for indirect customers, recent and pinned views in the Cost analysis preview, 3 new preview features, 7 new ways to save, 1 video, and 6 doc updates.

6 days, 11 hours ago @ azure.microsoft.com
Microsoft named a Leader in 2023 Gartner® Magic Quadrant™ for Integration Platform as a Service, Worldwide
Microsoft named a Leader in 2023 Gartner® Magic Quadrant™ for Integration Platform as a Service, Worldwide

We are excited that for the fifth consecutive time Gartner has positioned Microsoft as a Leader in the Magic Quadrant for Enterprise Integration Platform as a Service.

1 week ago @ azure.microsoft.com
Lessons learned optimizing Microsoft’s internal use of Azure
Lessons learned optimizing Microsoft’s internal use of Azure

We're sharing our story of first migrating to Azure, and then of optimizing our usage of Azure to bring our costs down.

1 week ago @ azure.microsoft.com
Microsoft named a Leader in the IDC MarketScape: Worldwide General-Purpose Computer Vision AI Software Platform 2022 Vendor Assessment
Microsoft named a Leader in the IDC MarketScape: Worldwide General-Purpose Computer Vision AI Software Platform 2022 Vendor Assessment

I am thrilled to announce that Microsoft has been recognized as a Leader in the IDC MarketScape: Worldwide General-purpose Computer Vision AI Software Platform 2022 Vendor Assessment.

1 week ago @ azure.microsoft.com
Cloud to Edge for efficient, agile, and sustainable retail
Cloud to Edge for efficient, agile, and sustainable retail

Prebuilt, edge-to-cloud, retail-specific technology solutions can help retailers increase the value of their data, empower store associates, elevate customer shopping experiences, and enable real-time sustainable supply chains.

1 week, 6 days ago @ azure.microsoft.com
Azure Native New Relic Service: Full stack observability in minutes
Azure Native New Relic Service: Full stack observability in minutes

Azure Native New Relic Service makes it effortless for developers and IT administrators to monitor their cloud applications.

1 week, 6 days ago @ azure.microsoft.com
Azure Native Qumulo Scalable File Service provides seamless, secure data storage
Azure Native Qumulo Scalable File Service provides seamless, secure data storage

We are glad to announce today that the storage suite of Azure products has grown with the addition of Azure Native Qumulo Scalable File Service.

2 weeks ago @ azure.microsoft.com
From Teams to PowerPoint: 10 ways Azure AI enhances the Microsoft Apps we use everyday
From Teams to PowerPoint: 10 ways Azure AI enhances the Microsoft Apps we use everyday

Azure AI is driving innovation and improving experiences for employees, users, and customers in a variety of ways, from increasing workday productivity to promoting inclusion and accessibility. The…

2 weeks ago @ azure.microsoft.com
What's new in Azure Data & AI: Empowering retailers to streamline operations and accelerate time to value
What's new in Azure Data & AI: Empowering retailers to streamline operations and accelerate time to value

Let’s explore what’s new for Azure Data & AI this month.

2 weeks, 6 days ago @ azure.microsoft.com
General availability of Azure OpenAI Service expands access to large, advanced AI models with added enterprise benefits
General availability of Azure OpenAI Service expands access to large, advanced AI models with added enterprise benefits

With Azure OpenAI Service now generally available, more businesses can apply for access to the most advanced AI models in the world—including GPT-3.5, Codex, and DALL•E 2—backed by the trusted enterprise-grade capabilities and AI-optimized infrastructure of Microsoft Azure, to create cutting-edge applications.

3 weeks ago @ azure.microsoft.com