There's numerous ways to achieve the task of a "computer mouse", with the design I described yesterday being quite outdated. So what are the common designs today?
Most mice (or wireless "hamsters", lol!) shine a laser light askew at a surface to reveal any roughness it might have. An image sensor rapidly compares the last 2 frames to determine how far the previous has to be shifted to match the next.
Involves a lot of very clever arithmetic, & a "digital signal processor" to run it on!
Any interested mathematicians: Please clarify why this works?
An efficient means of computing this "image cross-correlation" task is to convert the 2 frames into frequency-space using a Fast-Fourier Transform. With little more than a component-wise multiply & conversion back to signal-space gives us rankings for how close of a match each shift is! All that's left is to iterate over them to find the top-ranked coordinate & sum into the counter registers queried by the main CPU.
Touchpads & touchscreens are essentially the same device electronically, just a touchscreen has a screen behind it leading to very different ergonomics & affordances. So to an experience designer...
Modern ones are typically covered in capacitors, with your finger draining the capacitance beneath it. Rapidly pulsing each row to see which columns detect capacitive/voltage drains allows us to support multitouch.
To start we need to detect 2D runs of capacitive drains as corresponding to the same finger, choosing the coordinate of the largest drain as its coordinate. If we can get consistent-enough numbers, that value could be reported as the "pressure". Prediction can help accuracy here...
Smartphones include a signal-processor in their SoC to handle this! In our hypothetical hardware the Arithmetic Core could readily fill this role...
Now that we've detected touches we need to interpret gestures!
4/6
@screwtape Possibly. I've done plenty of searches for this thread but failed to find details. For that stage all I can find, beyond vague generalities that it involves a DSP, is what the inputs & outputs are.
I know detecting 1D runs is trivial, & I doubt 2D runs are much harder...
To figure out what those touches mean we need to analyze their timing & movement to determine whether its a swipe, pinch, or rotation. Swipes are best detected by our Arithmetic Core, whereas pinches & rotations require a bit more computational power to handle the trigonometry. Which can be offered by our Layout Co-Processor. But that gets into scrolling...
How to handle these gestures can be decided globally (typical for touchpads) or by the hovered element (typical for touchscreens).
For an HTML/CSS viewer I'd interpret those gestures globally turning them into clicks, open-menu, scrolls, & text-selections on particular elements.
But pointer-inputs aren't the only means of interacting with a computer! I've described voice commands earlier. Or you can tab/shift+tab between all clickable "links", stable-sorted by "tabindex" property & restyling them to indicate their focus.
Or analyze layout to figure out where arrow keys should navigate to. Or...
@alcinnz typical touch interface inputs are still extremely noisy, even after signal processing in the hardware and OS driver. See https://github.com/google/ink-stroke-modeler for an example of a software-based solution to smoothed, low-latency drawing with a stylus.
@alcinnz
Convolution (= same as computing the correlation upside-down and backwards) in regular space is element-wise multiplication in Fourier space.
The transform into fourier space (discrete fourier transform) turns out to be doable with surprisingly few operations, so it's lower complexity to do a Fast Fourier Transform, element-wise multiply, and then do a Fast inverse Fourier Transform back to regular space than compute a correlation in regular space.
Add comment