Serving deep learning models on IoT edge devices faces considerable challenges because of their restricted computational capabilities. Variant Parallelism (VP) mitigates this by distributing lightweight model variants, but has two main limitations: (i) variant generation typically lacks a systematic approach to ensure diversity, and (ii) ensembles composed of identical-sized variants underperform compared to heterogeneous ensembles. To address these issues, we propose Variant-Improved Parallelism (VIP), which advances VP through two main innovations. First, VIP introduces a structured variant generation pipeline that leverages knowledge distillation and targeted model slicing to efficiently produce diverse variants. Second, it employs diversity-inducing methods, such as data augmentation and class-imbalanced training, to enable effective homogeneous ensembles, expanding VP’s utility in environments with similar device capabilities. Experiments on ResNet50 with CIFAR10, CIFAR100, and ImageNet-1K demonstrate that VIP achieves consistent accuracy gains of up to 10%, while preserving VP’s advantages in fault tolerance and communication efficiency, supporting scalable and adaptive edge AI deployments.

As Artificial Intelligence (AI) applications increasingly migrate to the cloud, selecting appropriate deployment strategies that ensure performance while minimizing cost has become a key challenge. This paper investigates the deployment of a Computer Vision workload in four representative strategies across the edge–cloud continuum. These include cloud-only, edge-only, and hybrid edge-cloud architectures, incorporating both serverless and server-based computing models. Furthermore, we investigate the feasibility of integrating advanced techniques such as Split Computing and Early Exiting within hybrid architectures. We conduct real-world experiments throughout Germany on AWS cloud Availability Zones from Amazon and edge Wavelength Zones within Vodafone infrastructure. For each deployment strategy, we measure latency, accuracy, and operational cost. Our key findings reveal a 40% average lower end-to-end latency on edge deployments. We also outline the trade-offs and application scenarios for each strategy, providing practical guidance for the efficient deployment of computer vision systems in cloud/edge environments.

This work presents a two-stage digital twin methodology for developing and validating Decentralized Federated Learning (DFL) algorithms on resource-constrained microcontrollers. The first stage, our simulation-based twin, enables rapid prototyping and algorithm exploration without hardware constraints, while the second stage, based on leveraging several hardware emulation instances in a containerized environment, provides hardware-aware validation under realistic conditions including network delays, resource limitations, and communication protocols. This approach bridges the critical gap between research and deployment, enabling performance analysis at a pace impractical with physical hardware alone. We demonstrate how this digital twin pipeline is essential for robust Machine Learning Operations (MLOps) in IoT environments, allowing for scalable, cost-effective testing of decentralized tiny ML. Our initial results across simulation, emulation, and a cluster of real ESP32-S3 microcontrollers show that our twins faithfully reproduce physical device behavior, making it a valuable framework for advancing tiny, decentralized AI.

We introduce Gist, a decentralized federated learning framework for tiny microcontrollers. Rather than considering all model parameters as equally important, we let devices get the “gist” of the updates. Our contribution has three pillars: (1) it segments model parameters by their importance, identifying the most impactful updates; (2) it shares the segments probabilistically, ensuring rapid propagation of important knowledge while maintaining model diversity; and (3) it aggregates updates using a success-based scheme, giving more weight to information from better-performing peers. We implement and validate Gist through various simulation experiments, realistic large-scale emulation, and deployment on a physical cluster of ESP32-S3 microcontrollers. Across three models and two tasks, Gist outperforms existing baselines, achieving higher accuracy and faster convergence, especially in larger networks consisting of hundreds of devices.

Reinforcement learning (RL) offers a promising, adaptive alternative to heuristic-based autoscaling, yet its practical adoption in production environments remains negligible. In this paper, we argue that this gap between promise and practice is caused by three systemic challenges that violate fundamental RL assumptions: (i) generalization failures under workload and system drift; (ii) orchestration interference that obscures causality; and (iii) unreliable, delayed metric feedback. We substantiate these claims through an empirical study of two PPO-based autoscalers on real-world and synthetic workloads, demonstrating how these factors lead to policy instability and performance degradation. Our findings reveal that these challenges collectively frame autoscaling as a Partially Observable Markov Decision Process. We conclude that robust RL-based autoscaling requires a paradigm shift from purely algorithmic solutions toward systems-aware designs that model the partial observability and non-stationarity inherent in service autoscaling.

Microservice autoscaling balances service quality and operational costs through heuristic-based solutions that often overlook robustness. We adopt an adversarial perspective to expose vulnerabilities in autoscaling by identifying a near-optimal, seconds-scale traffic-aware bi-level pulsing attack scheduler that significantly outperforms traditional uniform DoS and existing pulsing attacks. Our proposed attack, which we call WaveSurfer, can exploit benign traffic patterns to increase attack effectiveness by up to 5.3x. This work highlights the urgent need for robust autoscaling mechanisms that can withstand sophisticated adversarial patterns.

Microservices autoscaling is essential for dynamically adjusting resources to meet fluctuating workload demands to maintain service-level objectives (SLO), such as latency, while minimizing resource usage. However, the control logic of modern autoscalers is susceptible to exploitation. Assessing its performance requires more than passive monitoring; the rapid evolution of application development has outpaced the availability of observability tools to benchmark and identify corner cases in autoscaling configurations relative to an application’s behavior. In this work, we aim to address a critical yet underexplored question: Can we systematically identify adversarial inputs, i.e., traffic anti-patterns that disproportionately increase SLO violations or operational costs—or both? We propose an adversarial benchmarking framework designed to generate traffic patterns tailored for finding autoscaling anti-patterns. It evolves strategies based on real-time observability signals from both the application and infrastructure layers. This problem is inherently complex due to its large solution space. To address this, we introduce heuristics that relax the problem while leveraging multiple parallel systems, each paired with a local controller and optimizer. These controllers act as individual agents being managed by a global controller, asynchronously generate diverse traffic patterns while collectively optimizing toward a shared adversarial objective. We evaluate our framework on two applications and three optimization methods. Our preliminary empirical results illustrate its effectiveness and suggest a good degree of generalizability.

As machine learning (ML) models grow in complexity and scale, distributed deployment across multiple devices has become essential for ensuring performance and scalability. However, the dynamic nature of distributed ML, where models must be frequently retrained, partitioned, and updated, exposes severe limitations in the current de-facto container-based model deployment. Specifically, the layered architecture of container filesystems is not well-suited for handling fine-grained model updates and partitioned ML deployments, leading to inefficient rebuilds and long delays. In this paper, we present 2DFS, a novel two-dimensional filesystem that enables independent updates, caching, and distribution of ML model components. We design and develop a complete ecosystem, including a builder, registry, and cache hierarchy, to streamline the build and deployment processes of ML models leveraging 2DFS. Our comprehensive evaluation of 14 real-world ML models demonstrates that 2DFS achieves up to 56x faster build times, 25x better caching efficiency, while providing on-demand image partitioning with no overhead. 2DFS is fully OCI-compliant and integrates seamlessly with existing infrastructures and container workflows.

In the realm of cluster operation, continuously validating and optimizing the configuration requires access to accurate cluster behavioral models. Network Digital Twins (NDTs) have emerged as a paradigm to provide such accurate, live representations of network systems. To capture the live state, NDTs need to anticipate the cluster behavior in a faster than real-time manner. With increasingly complex clusters, classical NDTs relying on detailed handcrafted simulators become too slow to fulfill this task. Leveraging measurements from the actual system demonstrates the potential to create more highlevel, lightweight NDTs that are still fairly accurate. Nonetheless, the degree of abstraction required to create fast and accurate data-driven NDTs is not well understood. To address this, our work investigates the impact of different abstraction levels on modeling accuracy. We develop and compare three Network Digital Twins of a Kubernetes Cluster - a Twin based on a Handcrafted Simulator, a Decentralized Data-driven Twin, abstracting individual system components, and a Centralized Data-driven Twin, abstracting the system as a whole. Our results show that Data-driven Twins improve the performance prediction by 18-53% over the handcrafted one, with the Centralized Twin surpassing the Decentralized Twin in accuracy by 35% and speed by two orders of magnitude.

Thanks to OCI standardization, containers now describe a complete end-to-end environment for packaging, distributing, and deploying software and its dependencies. Unfortunately, the complexity of modern applications creates use cases that are not yet supported by this technology. For example, packaging machine learning model weights in containers requires extensive build times, reduces flexibility, and forces developers to seek alternative, custom solutions. Another example is platform-specific drivers and binaries, which are often included in container fat images but involve cache invalidations for every minor change. In our work, we extend the container layered structure with a new two-dimensional filesystem layer type, specifically designed for efficient handling of large data. The proposed layer type allows highly parallelized image builds and fine-grained layer caching, while also providing a mechanism for on-demand partitions. Such partitions enable developers to request a subset of the image, e.g., containing only a portion of a machine learning model or a specific set of drivers for a given architecture. The proposed implementation is fully OCI-compliant, allowing distribution of such customized images to any container runtime with no additional effort. Our implementation, called 2DFS, achieves 56x faster build times and 25x better caching efficiency compared to Docker, while providing on-demand image partitioning with no overhead.

TBA

Sub-byte quantization on popular vector ISAs suffers from heavy waste of vector as well as memory bandwidth. The latest methods pack a number of quantized data in one vector, but have to pad them with empty bits to avoid overflow to neighbours. We remove even these empty bits and provide full utilization of the vector and memory bandwidth by our data-layout/compute co-design scheme. We implemented FullPack on TFLite for Vector-Matrix multiplication and showed up to 6.7x speedup, 2.75x on average on single layers, which translated to 1.56−2.11x end-to-end speedup on DeepSpeech.

Two major techniques are commonly used to meet real-time inference limitations when distributing models across resource-constrained IoT devices: (1) model parallelism (MP) and (2) class parallelism (CP). In MP, transmitting bulky intermediate data (orders of magnitude larger than input) between devices imposes huge communication overhead. Although CP solves this problem, it has limitations on the number of sub-models. In addition, both solutions are fault intolerant, an issue when deployed on edge devices. We propose variant parallelism (VP), an ensemble-based deep learning distribution method where different variants of a main model are generated and can be deployed on separate machines. We design a family of lighter models around the original model, and train them simultaneously to improve accuracy over single models. Our experimental results on six common mid-sized object recognition datasets demonstrate that our models can have \(5.8-7.1{\times}\) fewer parameters, \(4.3-31{\times}\) fewer multiply-accumulations (MACs), and \({2.5-13.2{\times}}\) less response time on atomic inputs compared to MobileNetV2 while achieving comparable or higher accuracy. Our technique easily generates several variants of the base architecture. Each variant returns only \(2k\) outputs \(1 \le k \le ({\frac{1}{2}\#classes})\), representing Top-k classes, instead of tons of floating point values required in MP. Since each variant provides a full-class prediction, our approach maintains higher availability compared with MP and CP in presence of failure.

With the increasing number of internet of things devices and the ability to enhance data privacy using federated learning (FL), there is a growing interest in exploring FL on resource-constrained devices. Centralized FL uses a central server (CS), which can lead to privacy issues, poor scalability, and high communication costs. To address these challenges, decentralized federated learning (DFL) has been proposed, which allows FL without a CS. To keep the communication cost low in DFL the gossip protocol can be used, where random clients exchange their models. However, this may result in slower convergence and lower accuracy. To overcome these limitations, we present AdaStair, a static method, and AdaLoss, a dynamic method that adaptively adjusts the learning rate of each client in a decentralized manner. Additionally, we modify the communication protocol of GossipFL and add quantization to further reduce the communication cost. We demonstrate the robustness of these methods in both simulations and real-world evaluations using microcontroller units in a partial WiFi ad-hoc mesh network. Our experimental results show that AdaStair and AdaLoss can improve accuracy up to 19 % and reduce communication costs by up to 57 % using a quantization scheme compared to a segmented gossip method.

Due to changing demand patterns driven by technological advancements and the rise of new applications and services, the provisioning of heterogeneous workloads is a crucial component of the resource allocation problem. Traditional resource allocation strategies such as reactive autoscaling or prediction-based proactive solutions, fail to meet the desired performance goals when the underlying demand arrival pattern changes. In this paper, we present HYPA, which combines reactive and proactive components to autoscale pods in a Kubernetes environment. In contrast to previous approaches of hybrid autoscaling, HYPA automatically reacts to drifts in the request arrival pattern. Specifically, it updates the model of its proactive component when the prediction performance decreases. The evaluation in a simulation on a variety of real-world traces, spanning multiple days, demonstrates that HYPA improves upon existing purely reactive and purely proactive horizontal pod autoscalers.

Cluster orchestrators such as Kubernetes (K8s) provide many knobs that cloud administrators can tune to conFigure their system. However, different configurations lead to different levels of performance, which additionally depend on the application. Hence, finding exactly the best configuration for a given system can be a difficult task. A particularly innovative approach to evaluate configurations and optimize desired performance metrics is the use of Digital Twins (DT). To achieve good results in short time, the models of the cloud network functions underlying the DT must be minimally complex but highly accurate. Developing such models requires detailed knowledge about the system components and their interactions. We believe that a data-driven paradigm can capture the actual behavior of a network function (NF) deployed in the cluster, while decoupling it from internal feedback loops. In this paper, we analyze the HTTP load balancing function as an example of an NF and explore the data-driven paradigm to learn its behavior in a K8s cluster deployment. We develop, implement, and evaluate two approaches to learn the behavior of a state-of-the-art load balancer and show that Machine Learning has the potential to enhance the way we model NF behaviors.

We present a self-operating Kubernetes (K8s) cluster that uses digital twinning and machine learning to autonomously adapt its Horizontal Pod Autoscaler (HPA) to workload changes. The demo uses a digital twin of a K8s cluster to gather performance statistics and learn a model for the workload. With the model, the cluster autonomously adjusts HPA parameters for better performance. The demo illustrates this process and shows that the requested pod seconds decrease by ∼37%, while the request latency stays mostly unaffected.

Although prior art has demonstrated negligible accuracy drop in sub-byte quantization—where weights and/or activations are represented by less than 8 bits—popular SIMD instructions of CPUs do not natively support these datatypes. While recent methods, such as ULPPACK, are already using sub-byte quantization on general-purpose CPUs with vector units, they leave out several empty bits between the sub-byte values in memory and in vector registers to avoid overflow to the neighbours during the operations. This results in memory footprint and bandwidth-usage inefficiencies and suboptimal performance. In this paper, we present memory layouts for storing, and mechanisms for processing sub-byte (4-, 2-, or 1-bit) models that utilize all the bits in the memory as well as in the vector registers for the actual data. We provide compute kernels for the proposed layout for the GEMV (GEneral Matrix-Vector multiplication) operations between weights and activations of different datatypes (e.g., 8-bit activations and 4-bit weights). For evaluation, we extended the TFLite package and added our methods to it, then ran the models on the cycle-accurate gem5 simulator to compare detailed memory and CPU cycles of each method. We compare against nine other methods that are actively used in production including GEMLOWP, Ruy, XNNPack, and ULPPACK. Furthermore, we explore the effect of different input and output sizes of deep learning layers on the performance of our proposed method. Experimental results show 0.96-2.1x speedup for small sizes and 1.2-6.7x speedup for mid to large sizes. Applying our proposal to a real-world speech recognition model, Mozilla DeepSpeech, we proved that our method achieves 1.56-2.11x end-to-end speedup compared to the state-of-the-art, depending on the bit-width employed.

Despite essential efforts towards advanced wireless medical devices for regular monitoring of blood properties, many such devices are not available or not affordable for everyone in many countries. Alternatively using ordinary devices, patients ought to log data into a mobile health-monitoring manually. It causes several issues: (1) clients reportedly tend to enter unrealistic data; (2) typing values several times a day is bothersome and causes clients to leave the mobile app. Thus, there is a strong need to use now-ubiquitous smartphones, reducing error by capturing images from the screen of medical devices and extracting useful information automatically. Nevertheless, there are a few challenges in its development: (1) data scarcity has led to impractical methods with very low accuracy: to our knowledge, only small datasets are available in this case; (2) accuracy-availability tradeoff: one can execute a less accurate algorithm on a mobile phone to maintain higher availability, or alternatively deploy a more accurate and more compute-intensive algorithm on the cloud, however, at the cost of lower availability in poor/no connectivity situations. We present an ensemble learning algorithm, a mobile-cloud computing service architecture, and a simple compression technique to achieve higher availability and faster response time while providing higher accuracy by integrating cloud- and mobile-side predictions. Additionally, we propose an algorithm to generate synthetic training data which facilitates utilizing deep learning models to improve accuracy. Our proposed method achieves three main objectives: (1) \(92.1\%\) and \(97.7\%\) accuracy on two different datasets, improving previous methods by \(40\%\), (2) reducing required bandwidth by \(45{\times}\) with \(1\%\) drop in accuracy, (3) and providing better availability compared to mobile-only, cloud-only, split computing, and early exit service models.