Channel Title: Microsoft Research Channel Description: At Microsoft Research, we accelerate scientific discovery and technology innovation to empower every person and organization on the planet to achieve more. We do this by bringing together the best minds across diverse disciplines and backgrounds to take on the most pressing research challenges for Microsoft and for society. Our Research Lens We consider research directions through the lens of the positive impact we aspire to create with and for customers, communities, and all of society. <video> Data Formulator: Create Rich Visualization with AI iteratively Description: Data Formulator is a research prototype tool that enables users to iterate with AI to create rich data visualizations. Users define their goals through a blend of user interface interactions and natural language inputs, while the AI automatically generates and adjusts the code to transform data into charts. </video> <video> GASP: Gaussian Avatars with Synthetic Priors Description: Gaussian Splatting has changed the game for real-time photo-realistic rendering. One of the most popular applications of Gaussian Splatting is to create animatable avatars, known as Gaussian Avatars. Recent works have pushed the boundaries of quality and rendering efficiency but suffer from two main limitations. Either they require expensive multi-camera rigs to produce avatars with free-view rendering, or they can be trained with a single camera but only rendered at high quality from this fixed viewpoint. An ideal model would be trained using a short monocular video or image from available hardware, such as a webcam, and rendered from any view. To this end, we propose GASP: Gaussian Avatars with Synthetic Priors. To overcome the limitations of existing datasets, we exploit the pixel-perfect nature of synthetic data to train a Gaussian Avatar prior. By fitting this prior model to a single photo or video and fine-tuning it, we get a high-quality Gaussian Avatar, which supports 360-degree rendering. Our prior is only required for fitting, not inference, enabling real-time application. Through our method, we obtain high-quality, animatable Avatars from limited data which can be animated and rendered at 70fps on commercial hardware. </video> <video> Pretrainer's Guide to Training Data: Measuring Effects of Age, Domain Coverage, Quality, & Toxicity Description: Pretraining is the preliminary and fundamental step in developing capable language models (LM). Despite this, pretraining data design is critically under-documented and often guided by empirically unsupported intuitions. To address this, we pretrain 28 1.5B parameter decoder-only models, training on data curated (1) at different times, (2) with varying toxicity and quality filters, and (3) with different domain compositions. First, we quantify the effect of pretraining data age. A temporal shift between evaluation data and pretraining data leads to performance degradation, which is not overcome by finetuning. Second, we explore the effect of quality and toxicity filters, showing a trade-off between performance on standard benchmarks and risk of toxic generations. Our findings indicate there does not exist a one-size-fits-all solution to filtering training data. We also find that the effects of different types of filtering are not predictable from text domain characteristics. Lastly, we empirically validate that the inclusion of heterogeneous data sources, like books and web, is broadly beneficial and warrants greater prioritization. These findings constitute the largest set of experiments to validate, quantify, and expose many undocumented intuitions about text pretraining, which we hope will help support more informed data-centric decisions in LM development. </video> <video> Look Ma, no markers: holistic performance capture without the hassle Description: We tackle the problem of highly-accurate, holistic performance capture for the face, body and hands simultaneously. Motion-capture technologies used in film and game production typically focus only on face, body or hand capture independently, involve complex and expensive hardware and a high degree of manual intervention from skilled operators. While machine-learning-based approaches exist to overcome these problems, they usually only support a single camera, often operate on a single part of the body, do not produce precise world-space results, and rarely generalize outside specific contexts. In this work, we introduce the first technique for marker-free, high-quality reconstruction of the complete human body, including eyes and tongue, without requiring any calibration, manual intervention or custom hardware. Our approach produces stable world-space results from arbitrary camera rigs as well as supporting varied capture environments and clothing. We achieve this through a hybrid approach that leverages machine learning models trained exclusively on synthetic data and powerful parametric models of human shape and motion. We evaluate our method on a number of body, face and hand reconstruction benchmarks and demonstrate state-of-the-art results that generalize on diverse datasets. See the project page for more details and dataset download instructions: https://aka.ms/synthmocap </video> <video> AI for Business Transformation: Lessons from Healthcare Description: AI is already helping healthcare organizations drive innovation and efficiency gains. Peter Lee and Vijay Mital explore how Microsoft is helping, and how the lessons from healthcare could apply across many other industries. 0:00 – Introduction 4:35 – Beginnings of AI transformation 6:22 – Early lessons from healthcare 12:30 – The importance of trust 18:17 – How do we get started? 25:00 – Building rapport with AI 27: 31 – A spectacular rate of progress 30:53 – Is my business ripe for AI? 35:04 – The emergence of multimodal AI 40:30 – Connecting the dots for practical opportunities 41:37 – How is this generation of AI innovation like previous tech? Related: Azure Essentials Show | Essential AI for Business: https://aka.ms/EssentialAIForBusiness </video> <video> Hairmony: Fairness-aware hairstyle classification Description: We present a method for prediction of a person’s hairstyle from a single image. Despite growing use cases in user digitization and enrollment for virtual experiences, available methods are limited, particularly in the range of hairstyles they can capture. Human hair is extremely diverse and lacks any universally accepted description or categorization, making this a challenging task. Most current methods rely on parametric models of hair at a strand level. These approaches, while very promising, are not yet able to represent short, frizzy, coily hair and gathered hairstyles. We instead choose a classification approach which can represent the diversity of hairstyles required for a truly robust and inclusive system. Previous classification approaches have been restricted by poorly labeled data that lacks diversity, imposing constraints on the usefulness of any resulting enrollment system. We use only synthetic data to train our models. This allows for explicit control of diversity of hairstyle attributes, hair colors, facial appearance, poses, environments and other parameters. It also produces noise-free ground-truth labels. We introduce a novel hairstyle taxonomy developed in collaboration with a diverse group of domain experts which we use to balance our training data, supervise our model, and directly measure fairness. We annotate our synthetic training data and a real evaluation dataset using this taxonomy and release both to enable comparison of future hairstyle prediction approaches. We employ an architecture based on a pre-trained feature extraction network in order to improve generalization of our method to real data and predict taxonomy attributes as an auxiliary task to improve accuracy. Results show our method to be significantly more robust for challenging hairstyles than recent parametric approaches. Evaluation with taxonomy-based metrics also demonstrates the fairness of our method across diverse hairstyles. For more details and dataset download instructions see the project page: http://aka.ms/hairmony </video> <video> Estimating mental workload in a simulated flight task using optical f-NIRS signals Description: Speakers: Elisabeth Heremans Host: Ivan Tashev Accurately estimating cognitive workload is essential for adaptive learning in pilot training, allowing flight simulation tasks to be tailored to the pilot's skill level. In this project, we investigate the use of functional near-infrared spectroscopy (f-NIRS) to measure cognitive workload in pilots during a simulated flight task. f-NIRS, a non-invasive brain imaging technique, monitors changes in blood oxygenation in the cerebral cortex. Participants performed a series of flight maneuvers varying in complexity, whilst f-NIRS, breathing, and electrocardiography (ECG) signals were measured. Workload levels were quantified using both objective (task performance metrics) and subjective (self-reported workload) measures. To predict objective workload levels from the f-NIRS signals, we discarded noisy signals, extracted useful features from the signals, and fed these to machine learning predictors. Our predictor performs mental workload estimation in an objective and subject-independent manner. Ultimately, we demonstrate the viability of fNIRS as a tool for real-time workload monitoring in aviation, with a potential application in adaptive pilot training. </video> <video> Quantum Lattice Enumeration in Limited Depth, Fernando Virdia Description: Speakers: Fernando Virdia Host: Michael Naehrig In 2018, Aono, Nguyen, and Shen (ASIACRYPT 2018) proposed to use quantum backtracking algorithms (Montanaro, TOC 2018; Ambainis and Kokainis, STOC 2017) to speedup lattice point enumeration. Aono et al.’s work argued that quantum walk speedups can be applied to lattice enumeration, achieving at least a quadratic asymptotic speedup à la Grover search while not requiring exponential amounts of quantum accessible classical memory, as it is the case for quantum lattice point sieving. In this talk, we will explore how to lower bound the cost of using Aono et al.’s techniques on lattice enumeration with extreme cylinder pruning, assuming a limit to the maximum depth that a quantum computation can achieve without decohering, with the objective of better understanding the practical applicability of quantum backtracking in lattice cryptanalysis. </video> <video> Low latency carbon budget 2023 Description: In 2023, global land carbon sinks—the Earth’s forests, soils, and vegetation that absorb CO2—experienced a dramatic decline, as revealed by a study published in National Science Review by several international research teams. Using dynamic global vegetation models, satellite fire emissions, OCO-2 satellite measurements, and ocean model emulators, the study provides a fast-track carbon budget for 2023, identifying unprecedented weakening of land carbon sinks. Publication: https://academic.oup.com/nsr/advance-article/doi/10.1093/nsr/nwae365/7831649 </video> <video> A Closer Look at Falcon Description: Speakers: Jonas Janneck Host: Melissa Chase Falcon is a winner of NIST’s six-year post-quantum cryptography standardization competition. Based on the celebrated full-domain-hash framework of Gentry, Peikert and Vaikuntanathan (GPV) (STOC’08), Falcon leverages NTRU lattices to achieve the most compact signatures among lattice-based schemes. Its security hinges on a Renyi divergence-based argument for Gaussian samplers, a core element of the scheme. However, the GPV proof, which uses statistical distance to argue closeness of distributions, fails when applied naively to Falcon. Additional implementation-driven deviations from the GPV framework further invalidate the original proof, leaving Falcon without a security proof despite its selection for standardization. In this talk, I want to give an overview of our results which demonstrate that introducing a few minor, conservative modifications allows for the first formal proof of the scheme in the random oracle model. At the heart of our analysis lies an adaptation of the GPV framework to work with the Renyi divergence, along with an optimized method for parameter selection under this measure. Furthermore, we obtain a provable version of the GPV framework over NTRU rings. Unfortunately, our analysis shows that despite our modification of Falcon-512 and Falcon-1024 we do not achieve strong unforgeability for either scheme. For plain unforgeability we are able to show that our modifications to Falcon-512 barely satisfy the claimed 120-bit security target and for Falcon-1024 we confirm the claimed security level. </video> <video> Improving the Security of United States Elections with Robust Optimization Description: For more than a century, election officials across the United States have inspected voting machines before elections using a procedure called Logic and Accuracy Testing (LAT). This procedure consists of election officials casting a test deck of ballots into each voting machine and confirming the machine produces the expected vote total for each candidate. In this talk, I will bring a scientific perspective to LAT by introducing the first formal approach to designing test decks with rigorous security guarantees. Specifically, we propose using techniques from operations research and mathematical optimization to find test decks that are guaranteed to detect any voting machine misconfiguration that would cause votes to be swapped across candidates. Out of all the test decks with this security guarantee, our approach yields the test deck with the minimum number of ballots, thereby minimizing implementation costs for election officials. To facilitate deployment at scale, we developed a practical exact algorithm for solving our optimization problem based on mixed-integer programming and the cutting plane method. In partnership with the Michigan Bureau of Elections, we retrospectively applied our approach to all 6928 ballot styles from Michigan's November 2022 general election; this retrospective study reveals that the test decks with rigorous security guarantees obtained by our approach require, on average, only 1.2% more ballots than current practice. Our approach has since been piloted in real-world elections by the Michigan Bureau of Elections as a low-cost way to improve election security and increase public trust in democratic institutions. This is joint work with Alex Halderman (Michigan) and Braden Crimmins (Stanford and Michigan), and our preprint is available at https://arxiv.org/pdf/2308.02306.pdf </video> <video> AI for Business Transformation: Multimodal Models Description: Multimodal models hold the key to unprecedented advances in AI, moving beyond text and numbers to incorporate a much wider spectrum of inputs. AI research leaders Peter Lee and Vijay Mital discuss how researchers are exploring a new generation of models to open broad new opportunities for businesses. 0:00 – Introduction 7:39 – Different types of AI models 12:46 – The emergence of multimodal AI 17:17 – Large action models 21:42 – AI apps 30:06 – The transformation mindset Related: Azure Essentials Show | Essential AI for Business: https://aka.ms/EssentialAIForBusiness </video> <video> Enhancing Security of Bluetooth Secure Connections via Deferrable Authentication Description: Speakers: Olga Sanina Host: Kim Laine The Bluetooth protocol for wireless connection between devices comes with several security measures toprotect confidentiality and integrity of data. At the heart of these security protocols lies the SecureSimple Pairing, wherewith the devices can negotiate a shared key before communicating sensitivedata. Despite the good intentions, the Bluetooth security protocol has repeatedly been shown tobe vulnerable, especially with regard to active attacks on the Secure Simple Pairing. In the talk, we present a mechanism to limit active attacks on the Secure Connections protocol (the more secure version of the Secure Simple Pairing protocol), without infringing on the current Bluetooth protocol stack specification. The idea is to run an authentication protocol, like a classical challenge-response step for certified keys, within the existing infrastructure, even at a later, more convenient point in time. Not only does this authentication step ensure freshness of future encryption keys, but an interesting feature is that it—a posteriori—also guarantees security of previously derived encryption keys. This approach prevents a large set of known attacks on the Bluetooth protocol. </video> <video> Future Directions for XR Interactions with Advanced Sensing Techniques and Haptic Design Frameworks Description: Speakers: Sang Ho Yoon Host: Michel Pahud The advancement in sensor/actuator hardware has enabled the development of wearable interfaces for AR/VR beyond. However, understanding user intent and providing appropriate feedback remain significant challenges for us when going beyond HMD or glasses for XR interactions. To address this, I have explored methods to unlock the capability of understanding the user’s intents. Additionally, I have explored haptic-based interaction to offer tailored feedback for AR/VR experiences. In this talk, I will present our lab’s research on leveraging biosignals to offer novel input techniques such as context-aware hand gesture, hand pressure estimation, and continuous input with microgestures. I will then present multiple haptic interaction frameworks that fulfills the gap between human perception and intended interaction design to promote the integration of haptic feedback into AR/VR environments for HCI practitioners and end users. </video> <video> Introducing BiomedParse, a groundbreaking foundation model for biomedical image analysis Description: Image analysis is fundamental for clinical diagnostics and biomedical discovery. In this video, we introduce BiomedParse, a biomedical foundation model for holistic image analysis that can jointly conduct recognition, detection, and segmentation for 64 major object types across 9 imaging modalities. BiomedParse adopts a novel image parsing framework for joint learning across the three interdependent tasks. Users can conduct image analysis simply by specifying objects through a unified text prompt. To pretrain BiomedParse, we harness GPT-4 for data synthesis from existing segmentation datasets and created the first dataset for biomedical image parsing, comprising over six million triples of image, object description, and segmentation mask. BiomedParse substantially outperforms prior state-of-the-art methods, often by large margin, especially for the most challenging complex and irregular-shaped objects. BiomedParse is an all-in-one tool for biomedical image analysis on all major image modalities, paving the way for efficient and accurate image-based biomedical discovery. Real-world Evidence group: https://www.microsoft.com/en-us/research/group/real-world-evidence/ </video>

Contacts and Links

Description

At Microsoft Research, we accelerate scientific discovery and technology innovation to empower every person and organization on the planet to achieve more. We do this by bringing together the best minds across diverse disciplines and backgrounds to take on the most pressing research challenges for Microsoft and for society.

Our Research Lens
We consider research directions through the lens of the positive impact we aspire to create with and for customers, communities, and all of society.

Videos from channel