[SDL] tvOS swipe left/right gesture support (patch enclosed)

[oviano]

I found that events weren't being generated for the simple directional swipes on the Apple TV touch remote.

While it's possible to use the touch press/release/motion events to detect simple swipes of this nature (and I had this working well enough) it struck me that this was re-inventing the wheel.

As such, I have written a patch that recognises the proper swipe up/down/left/right events issued in UIKit and passes these on as directional key presses. This also means that the touch remote operates the same way as the old-style IR remote in this respect.


Code:

diff -r 007dfe83abf8 src/video/uikit/SDL_uikitview.m
--- a/src/video/uikit/SDL_uikitview.m Wed Oct 19 20:50:33 2016 -0700
+++ b/src/video/uikit/SDL_uikitview.m Sun Dec 18 14:57:42 2016 +0300
@@ -48,7 +48,20 @@
#if !TARGET_OS_TV
self.multipleTouchEnabled = YES;
#endif
-
+#if TARGET_OS_TV
+ UISwipeGestureRecognizer *swipeUp = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+ swipeUp.direction = UISwipeGestureRecognizerDirectionUp;
+ [self addGestureRecognizer:swipeUp];
+ UISwipeGestureRecognizer *swipeDown = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+ swipeDown.direction = UISwipeGestureRecognizerDirectionDown;
+ [self addGestureRecognizer:swipeDown];
+ UISwipeGestureRecognizer *swipeLeft = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+ swipeLeft.direction = UISwipeGestureRecognizerDirectionLeft;
+ [self addGestureRecognizer:swipeLeft];
+ UISwipeGestureRecognizer *swipeRight = [[UISwipeGestureRecognizer alloc] initWithTarget:self action:@selector(swipeGesture:)];
+ swipeRight.direction = UISwipeGestureRecognizerDirectionRight;
+ [self addGestureRecognizer:swipeRight];
+#endif
touchId = 1;
SDL_AddTouch(touchId, "");
}
@@ -56,6 +69,27 @@
return self;
}

+#if TARGET_OS_TV
+- (void)swipeGesture:(UISwipeGestureRecognizer *)sender
+{
+ switch (sender.direction)
+ {
+ case UISwipeGestureRecognizerDirectionUp:
+ SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_UP);
+ break;
+ case UISwipeGestureRecognizerDirectionDown:
+ SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_DOWN);
+ break;
+ case UISwipeGestureRecognizerDirectionLeft:
+ SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_LEFT);
+ break;
+ case UISwipeGestureRecognizerDirectionRight:
+ SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_RIGHT);
+ break;
+ }
+}
+#endif
+
- (void)setSDLWindow:(SDL_Window *)window
{
SDL_WindowData *data = nil;

[Alex Szpakowski]

SDL is already supposed to send left/right/up/down key presses when “press” events are detected on the Apple TV Remote ( https://developer.apple.com/library/content/documentation/General/Conceptual/AppleTV_PG/DetectingButtonPressesandGestures.html <https://developer.apple.com/library/content/documentation/General/Conceptual/AppleTV_PG/DetectingButtonPressesandGestures.html> ) – does that work for you? Does your extra gesture recognizer code interfere with that at all?

Post by oviano
I found that events weren't being generated for the simple directional swipes on the Apple TV touch remote.
While it's possible to use the touch press/release/motion events to detect simple swipes of this nature (and I had this working well enough) it struck me that this was re-inventing the wheel.
As such, I have written a patch that recognises the proper swipe up/down/left/right events issued in UIKit and passes these on as directional key presses. This also means that the touch remote operates the same way as the old-style IR remote in this respect.
diff -r 007dfe83abf8 src/video/uikit/SDL_uikitview.m
--- a/src/video/uikit/SDL_uikitview.m Wed Oct 19 20:50:33 2016 -0700
+++ b/src/video/uikit/SDL_uikitview.m Sun Dec 18 14:57:42 2016 +0300
@@ -48,7 +48,20 @@
#if !TARGET_OS_TV
self.multipleTouchEnabled = YES;
#endif
-
+#if TARGET_OS_TV
+ swipeUp.direction = UISwipeGestureRecognizerDirectionUp;
+ [self addGestureRecognizer:swipeUp];
+ swipeDown.direction = UISwipeGestureRecognizerDirectionDown;
+ [self addGestureRecognizer:swipeDown];
+ swipeLeft.direction = UISwipeGestureRecognizerDirectionLeft;
+ [self addGestureRecognizer:swipeLeft];
+ swipeRight.direction = UISwipeGestureRecognizerDirectionRight;
+ [self addGestureRecognizer:swipeRight];
+#endif
touchId = 1;
SDL_AddTouch(touchId, "");
}
@@ -56,6 +69,27 @@
return self;
}
+#if TARGET_OS_TV
+- (void)swipeGesture:(UISwipeGestureRecognizer *)sender
+{
+ switch (sender.direction)
+ {
+ SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_UP);
+ break;
+ SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_DOWN);
+ break;
+ SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_LEFT);
+ break;
+ SDL_SendKeyboardKey(SDL_PRESSED, SDL_SCANCODE_RIGHT);
+ break;
+ }
+}
+#endif
+
- (void)setSDLWindow:(SDL_Window *)window
{
SDL_WindowData *data = nil;
_______________________________________________
SDL mailing list
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org <http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org>

[oviano]

That's great, but it doesn't work and no such events come through.

If you can point me in the direction of where the directional swipe gestures are handled in the code I can try and fix it, but I couldn't see it.

If you're talking about the old (non touch) remote then yes that generates the events correctly.

I'm talking about the swipes on the touch remote.

[Alex Szpakowski]

I just tested on the tvOS Simulator and the key press events work for me with the latest SDL code from hg.

https://hg.libsdl.org/SDL/file/f6cd81aab88e/src/video/uikit/SDL_uikitview.m#l204 <https://hg.libsdl.org/SDL/file/f6cd81aab88e/src/video/uikit/SDL_uikitview.m#l204>

Post by oviano
That's great, but it doesn't work and no such events come through.
If you can point me in the direction of where the directional swipe gestures are handled in the code I can try and fix it, but I couldn't see it.
If you're talking about the old (non touch) remote then yes that generates the events correctly.
I'm talking about the swipes on the touch remote.
_______________________________________________
SDL mailing list
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org <http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org>

[oviano]

Ok I'll try the simulator later but you're testing the swipe gestures, correct?

[Alex Szpakowski]

The Apple TV Remote touchpad “tap” actions, as described here: https://developer.apple.com/tvos/human-interface-guidelines/remote-and-controllers/#buttons <https://developer.apple.com/tvos/human-interface-guidelines/remote-and-controllers/#buttons>

The ‘tap’ actions map pretty cleanly to button presses, in terms of what their behaviour describes. SDL doesn’t really have analogous user-facing APIs that describe OS-handled swipe gestures in the same manner, however you can access the Remote’s touchpad as joystick axes in order to get a more granular description of what the user is doing with it.

Post by oviano
Ok I'll try the simulator later but you're testing the swipe gestures, correct?
_______________________________________________
SDL mailing list
http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org <http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org>

[oviano]

The up/down/left/right swipe gestures on Apple TVs are the primary method to navigate around button/menu systems on the device so assuming that some SDL apps will have button or menu systems that require the user to navigate from one button to another then it's far better to hook into these gestures...

Otherwise you are expecting the user to re-implement these themselves and inevitably there will be subtle differences in "feel" compared to the official ones leading to a substandard user experience.

[Alex Szpakowski]

Given the fact that key presses can’t represent swipe strength which is one of the main purposes of a swipe

Actually I suppose the Apple documentation is a bit misleading about this. :)

[Alex Szpakowski]

I pushed a commit which should do what you want: https://hg.libsdl.org/SDL/rev/a6babd973955

Post by Alex Szpakowski

Given the fact that key presses can’t represent swipe strength which is one of the main purposes of a swipe

Actually I suppose the Apple documentation is a bit misleading about this. :)

[oviano]

Well the swipe gesture I'm talking about hooking into, and which my patch uses is just a simple standard swipe with direction only. There is concept of strength:

https://developer.apple.com/reference/uikit/uiswipegesturerecognizer

So it maps pretty easily to the arrow key presses.

It's funny - I actually had no idea (despite owning a touch-based Apple TV since they were available) that you could tap to produce the direction "buttons", I'd always swiped which is why it seems to me such a big omission. I simply had no way to navigate within the UI of my application (a video player) :)

[oviano]

Thanks for making the change - I haven't tested your slight variation on my code yet but I will at some point and let you know if I encounter any problems.