[ANSWERED] Screen Capture & HID interactivity

[eamon]

Hi all

I have a set up using the screen capture alpha actor and joystick for an interactive website. Suprisingly enough, I have a query re how best to make this work.

I am running the website in Chrome on the 1st monitor and using Screen Capture Alpha to capture this screen and project on the second screen/show projector. This works fine and I have used izzy map to bring it into position.

The website is an interactive one where the HID/joystick is used to play a game. I am wondering if you can interact with the captured website via the HID. I only had a chance to start playing when I was pulled away for other stuff but it does not seem like a logical thing to be able to do in my set up

Is it possible it interact with the projected image as if I was on the actual website and play the game? I have to read up on the how to for HID work with isadora but I am wondering is this something that is possible or am I barking up the wrong tree?

The joystick is not perfect (read needs more for the interactive game) but I would move to an OSC/touch OSC customised layout but the primacy of my query re the interactivity with a captured screen reads the same for me?

Is this is not possible, can I ask that it becomes a feature request?

My current set up is

- Mac Mini running OS Catalina

- Izzy 3.08 f12

- I have just seen 3.1.1 so I have yet to try it out.

I would be grateful if you could point in another direction if this is non-runner!

eamon

[DusX]

@eamon

I think I understand what it is you want to do to some extent. It seems you want to use the HID device to both control the Isadora screen Capture, and simultaneously interact with elements on the website.

This isn't possible within Isadora. What you would need to do is use some scripting language (python, maybe Processing) to control the systems Mouse. In this way you could relay information to the scripting language which then in turn controls the mouse (position, mouse down etc)

I would think that sending out OSC from Isadora that echos/relays the HID input (or the touchOSC data) would be the easier way to make this work. I know that Python offers a few modules for this type of system control.
This python module looks promising for mouse control: https://pypi.org/project/pynpu...

With Apples tightening of security in macOS there may be some hurdles to tackle.

[bonemap]

@eamon

Hi,

 Have played around with getting website control embedded in an Isadora patch and did get some interactivity with webpages by developing an FFGL browser plugin. The plugin is a prototype based on the attributes available for web interactivity provided by Vuo software and FFGL plugin export that Vuo provides. I have a demonstration patch that might be of interest here:
https://www.dropbox.com/sh/lthh8t8ehw795kv/AADr_avr2w-o8ZAdCTYrQnmNa?dl=0

The plugin has x,y inputs for rudimentary cursor interaction with a live web page. But there are plenty of limitations in regard to what content will display and work correctly.

My demonstration patch uses an API from Archive.org to do content search and the FFGL plugin allows a cursor selection of content links. Your web interface sounds more layered and may not even work with the plugin. It is a good idea to have a look at the read me file that is included in the package. If you do find something useful and want to ask any questions let me know.

Best Wishes

Russell

[CitizenJoe]

@bonemap said:

... provided by Vuo software and FFGL plugin export that Vuo provides.

 For future users, note that this is MAC only.

Cheers,

Hugh

[bonemap]

@citizenjoe

Thanks Hugh, I should have indicated that up front. 

Best wishes

Russell