Ubuntu Frame: Better Cursor Control For Touchscreens

When you're building applications that run on a variety of touchscreens, from small 10-inch displays to larger 32-inch panels, user experience is paramount. Recently, we've encountered some feedback regarding the visibility of the mouse cursor within our Ubuntu Frame applications. While Ubuntu Frame is a fantastic tool for displaying data and running applications, the current limitations around cursor management, especially on touch-enabled devices, have led to a few usability hiccups. We're exploring ways to enhance this, aiming for a smoother, more intuitive interaction for our users. Our goal is to ensure the cursor behaves as expected, disappearing when not needed and appearing only when genuinely useful, thereby improving the overall presentation and interaction with our touch screen interfaces.

The Cursor Conundrum: When "Auto" Isn't Enough

One of the primary issues we're facing revolves around the cursor=auto setting in Ubuntu Frame. While this is a great starting point – it correctly hides the cursor at startup, which is ideal for our informational displays – the problem arises once any mouse activity occurs. As soon as a mouse is moved, the cursor becomes permanently visible. For many of our use cases, the screen is intended to function as an unattended information kiosk after an initial setup or interaction. Users might use a mouse to click an initial link or enter credentials, and after that, the screen should ideally remain clean and free of extraneous on-screen elements like a persistent cursor. This is where we believe an inactivity-based cursor hiding mechanism would be a significant improvement. We've seen this implemented effectively in other environments, such as Sway, with its hide_cursor 3000 setting, which hides the cursor after 3000 milliseconds (3 seconds) of inactivity. Implementing a similar feature in Ubuntu Frame would allow the cursor to naturally disappear after a user has finished their interaction, restoring the clean look of the display until the next input is required. This simple addition could dramatically improve the aesthetic and functional experience for users interacting with our touch screen applications.

It's important to understand that our applications are often deployed in public-facing or operational environments where direct, continuous mouse interaction isn't the norm. After the initial setup, login, or selection of a specific data feed, the screen is meant to operate autonomously, presenting information without distraction. A visible cursor, even if static, can be perceived as a visual anomaly or even a point of confusion for users who are primarily interacting via touch. Therefore, the ability for the cursor to gracefully hide itself after a period of no mouse activity is not just a cosmetic preference; it's a functional requirement for creating a polished and professional user interface on touch-enabled devices. This feature would align Ubuntu Frame with modern UI/UX principles, ensuring that the interface adapts intelligently to the user's interaction patterns.

Furthermore, the implementation of such a feature doesn't necessarily need to be complex. It could be a simple configuration option within Ubuntu Frame itself, allowing administrators or developers to set a timeout value. This timeout would trigger the hiding of the cursor after a specified duration of mouse inactivity. The beauty of this approach is its subtlety; users wouldn't need to actively do anything to hide the cursor. It would simply fade away, leaving the screen uncluttered. When a user does need to interact with the screen again using a mouse, a simple movement would instantly bring the cursor back, providing immediate feedback and control. This dynamic behavior is crucial for touch screen applications that need to balance periods of passive display with moments of active user input, making the overall experience feel more responsive and less intrusive.

We believe that integrating this kind of intelligent cursor management would significantly elevate the user experience on touch screen devices running Ubuntu Frame. It addresses a common point of friction and moves us closer to a truly seamless integration of applications within these specialized hardware environments. The benefits extend beyond just aesthetics; a clean, uncluttered screen can lead to better information absorption and a more focused user interaction, especially in environments where attention spans might be limited or the primary goal is information dissemination.

Touch vs. Mouse: Differentiating Input Events

Another crucial aspect of our touch screen deployment involves distinguishing between touch events and actual mouse events. Currently, any touch event on the screen, regardless of where the touch occurs, results in the cursor appearing in the top-left corner of the screen. This behavior is not ideal because it can be misleading. For instance, if a user touches the middle of the screen to interact with an element there, the cursor appearing far away in the corner doesn't accurately represent the point of interaction. What we ideally need is the capability for the mouse cursor to be visible only after genuine mouse events, not touch events. However, we also need to retain the ability to operate these devices using a mouse, particularly during the initial setup or login phases. This means we can't simply disable the mouse cursor entirely based on the presence of touch input.

The challenge lies in finding a balance. We need a system that is sensitive to touch for the primary interaction method but still allows for precise mouse control when necessary. A possible solution could involve a configuration that prioritizes touch input for cursor visibility. When a touch occurs, the system would register the touch point for interaction but would not necessarily make the mouse cursor appear. Instead, the cursor would only materialize upon a subsequent mouse movement. This would ensure that the cursor only appears when the user is actively intending to use the mouse, rather than as a byproduct of a touch event. This separation of concerns between touch and mouse input would lead to a much cleaner and more intuitive user interface on touch-enabled devices.

Consider a scenario where an application is running on a large touch screen. A user might tap on a button located in the bottom right corner. With the current behavior, the cursor might pop up in the top left, unrelated to the user's action. If we could configure Ubuntu Frame to ignore touch events for cursor visibility, the cursor would remain hidden. If, subsequently, the user picks up a physical mouse and moves it to select an option, then the cursor would appear, accurately reflecting the mouse's position. This nuanced approach respects the primary interaction method (touch) while ensuring that the secondary method (mouse) is still fully functional and represented accurately when used.

We recognize that some applications or setup routines might still rely on mouse input. Therefore, a flexible configuration is key. Perhaps there could be a setting that allows enabling mouse cursor visibility triggered by touch only during specific application contexts or system states (e.g., during the login screen). Once the user is past that initial phase, the cursor would revert to only appearing on mouse movement. This kind of conditional logic would provide the best of both worlds: seamless touch interaction without unwanted cursor artifacts, and full mouse functionality when it's needed and expected. The ability to fine-tune this behavior based on the specific needs of the application and deployment environment is what we are aiming for.

This proposed distinction between touch and mouse input for cursor visibility is crucial for developing robust and user-friendly applications on touch screen hardware. It directly addresses the user complaints we've received and aligns with best practices for modern touch interface design. By separating the visual feedback for touch actions from the appearance of the mouse cursor, we can create a more intuitive and less cluttered experience for our users.

Under the Hood: Frame or Mir?

This brings us to the technical question: is this cursor behavior managed by the Frame binary itself, or is it a function of the Mir compositor? Understanding where these cursor control functionalities reside is critical for us to determine the best path forward for implementing these improvements. If the cursor hiding and touch-event-based visibility are configurable within the Mir compositor, then we would focus our efforts on Mir's configuration options or potential patches. Mir, as the display server and compositor, is responsible for managing display surfaces, input events, and visual elements, including the cursor. Therefore, it's highly plausible that Mir has the underlying mechanisms to handle these nuances of cursor behavior based on different input types and inactivity timers.

Alternatively, if Ubuntu Frame, as the application or window manager layer, has specific hooks or settings that dictate how it interacts with the compositor regarding cursor management, then the solution might lie within Frame's configuration or API. Ubuntu Frame acts as a bridge, often embedding applications and managing their display within a Wayland environment powered by Mir. It's possible that Frame itself interprets input events and instructs Mir on how to present the cursor, or it might have its own logic for managing cursor visibility independent of Mir's direct control, relying on Mir only for the rendering of the cursor itself. The cursor=auto setting suggests a level of control exerted by Ubuntu Frame.

To effectively address the issues of inactivity-based hiding and differentiating touch from mouse events, we need to pinpoint the component responsible. Investigating Mir's documentation, particularly sections related to input handling, cursor management, and Wayland protocols, would be a logical next step. Similarly, examining the configuration options and internal workings of Ubuntu Frame would shed light on its role in this process. It's possible that the desired functionality requires a combination of configurations in both Mir and Ubuntu Frame, or perhaps a modification to one or both components. The ultimate goal is to achieve a flexible and intelligent cursor behavior that enhances the user experience on touch screens, making the system feel more responsive and less intrusive. We are eager to explore the technical underpinnings to implement these much-needed improvements.

Conclusion: Towards a Smoother Touch Experience

Enhancing cursor hiding and management on touch screens within Ubuntu Frame is a key step towards creating more polished and user-friendly applications. By implementing inactivity-based cursor hiding and intelligently differentiating between mouse and touch events, we can significantly improve the user experience on a wide range of touch-enabled devices. We are keen to collaborate with the Ubuntu Frame and Mir development communities to explore the feasibility and implementation of these features. Ultimately, our aim is to ensure that Ubuntu Frame provides a seamless and intuitive interface, whether used with a mouse or touch, in any deployment scenario.

For further insights into Wayland compositors and their capabilities, you might find the Wayland Project documentation to be an invaluable resource.