Sensomusic

x.iso · 2018-09-28 22:21:40

Suddenly idea came to mind when I remembered how in professional TV studios you can find oscilloscopes measuring RBG signal from video or HSL or other parameters derived from the current frame of the video. Shouldn't be technically a problem to do the same in digital, but is it possible to get any wavefrom derived from video (or bitmap) out with the tools Usine have? I suspect it should be with bitwise conversions and math modules, but I'm not really good at that stuff.

In the end the signals could be used as oscillators for synths or LFO's and it could get really interesting from there.

oli_lab · 2018-09-29 09:53:59

very interesting.
could you point us towards such hardware oscilloscopes ?

23fx23 · 2018-09-29 10:09:26

there are converters in \color, usine color to separated rgb, or hsl and vice versa, and in video\ video to color array,
could scan a line, find grid median, build and histogram or something with that.

but the main question i think is how to translate that to a continuous sound waveform. a typical video framerate is only 25fps, vs 44100 needed audio,
there would need something to generate blocs size arrays that have a making sense 'continuity' from one frame to the next this sound more tricky reflexion
but very interesting.

x.iso · 2018-09-29 22:28:49

23fx23 wrote:

there are converters in \color, usine color to separated rgb, or hsl and vice versa, and in video\ video to color array,
could scan a line, find grid median, build and histogram or something with that.

but the main question i think is how to translate that to a continuous sound waveform. a typical video framerate is only 25fps, vs 44100 needed audio,
there would need something to generate blocs size arrays that have a making sense 'continuity' from one frame to the next this sound more tricky reflexion
but very interesting.

well that's not really much of a problem since you can just use Smooth to blend choppy data.

x.iso · 2018-09-29 22:50:33

oli_lab wrote:

very interesting.
could you point us towards such hardware oscilloscopes ?

https://www.youtube.com/watch?v=PQQZlSyF2-0
here's typical example how it looks

oli_lab · 2018-09-29 23:28:28

x.iso wrote:

oli_lab wrote:
very interesting.
could you point us towards such hardware oscilloscopes ?

https://www.youtube.com/watch?v=PQQZlSyF2-0
here's typical example how it looks

Ok
what you see is the analog signal of an old CRT TV.
it is sync to the signal that make the spot going from bottom rigtnt to top left at the end of each frame.
I'm not sure we get similar signals with digital video.

will need to scan the entire content of the pixel array line by line
and asuming we managed to interpolate those datas to ply them at desired audiorate. we'll get probably something very noisy !

let say we have a 800x600 videostream
that is 480 000 pixels

Now we want to play this image at 100Hz, We need to read the entire image 100 times per second, that is (if we assume the sample rate is 48 000Hz) to read a sample that is of 48000/100 = 480 samples.
so we need to get the mean value of each subsequent 1000 pixels so the 480 000 pixel will be représented in the 480 audio samples. Or simply read every 1000 pixels in the array and interpolate

we need a new module !

x.iso · 2018-09-30 23:42:38

So yeah, with some patching I was able to get some waveforms derived from video, now I'm going to make some simple synth out of it.

x.iso · 2018-09-30 23:45:59

Actually, there's so much data in those arrays that I guess it can be chopped to chunks for independent voices

x.iso · 2018-10-01 02:27:53

here's results so far. looks promising
https://youtu.be/Eza6AtpbSlA

23fx23 · 2018-10-02 09:53:04

Cool! Gonna check tonight

x.iso · 2018-10-02 18:54:57

here's a patch I've made so far. You might need to swap VST limiter for stock one or something else.
http://www.sensomusic.org/forums/upload … dSynth.pat

Last edited by x.iso (2018-10-02 18:55:06)

x.iso · 2018-10-03 01:17:36

By the way, is it possible to 'smooth' array values? By that I mean lessen the gap between adjacent values by a factor, could help make smoother waveforms.

23fx23 · 2018-10-03 08:17:16

As far as i remember, not directly via build in modules. Could use a lowpass filter but that will limited to fixed blocs size. Sadly very buzy at work till this weekend, but if wanna investigate i had an addon that used a script procedure to blur/smooth an array. Its called automizer if still accessible in addons

x.iso · 2018-10-03 14:37:47

It works better with oscillator module from Oli_Lab. here's some progress, quite usable at this point.
https://www.youtube.com/watch?v=HPsv44YiLhs

oli_lab · 2018-10-03 16:07:06

x.iso wrote:

It works better with oscillator module from Oli_Lab. here's some progress, quite usable at this point.
https://www.youtube.com/watch?v=HPsv44YiLhs

I was about to tell you that.
I did some experiments from the picture of your patch.
With draw lut you don't need so much to compress the array.
I did not used the concat array but used separate oscillators for each color.

x.iso · 2018-10-03 18:12:46

oli_lab wrote:

x.iso wrote:
It works better with oscillator module from Oli_Lab. here's some progress, quite usable at this point.
https://www.youtube.com/watch?v=HPsv44YiLhs

I was about to tell you that.
I did some experiments from the picture of your patch.
With draw lut you don't need so much to compress the array.
I did not used the concat array but used separate oscillators for each color.

yeah, next thing I was going to try is to get 3 OSC's for each color and try modulations between them, then there's also option to modulate vertical VS horizontal arrays, HSL instead of RGB and so on. many options to try out.

99Letters · 2018-10-05 22:34:15

hello, can I ask you?

now some audio sounds into usinehollyhock,
but don't know why sounds can't inside to its??

ummmmm.something help me??
use motu828. and I can see audio coming ...but no sounds.
I watched this tutorial video.

can u help me??
http://www.sensomusic.org/forums/upload … .53.33.png

actually use Ableton 10
can sounds from audio interface through come inside 3/4 channel.
http://www.sensomusic.org/forums/upload … .56.01.png

and actually I put audio from Mac book pro inside microphone.
also cant come to stereo out too.

Last edited by 99Letters (2018-10-05 23:25:54)

x.iso · 2018-10-06 06:30:09

99Letters wrote:

hello, can I ask you?

now some audio sounds into usinehollyhock,
but don't know why sounds can't inside to its??

ummmmm.something help me??
use motu828. and I can see audio coming ...but no sounds.
I watched this tutorial video.

can u help me??
http://www.sensomusic.org/forums/upload … .53.33.png

actually use Ableton 10
can sounds from audio interface through come inside 3/4 channel.
http://www.sensomusic.org/forums/upload … .56.01.png

and actually I put audio from Mac book pro inside microphone.
also cant come to stereo out too.

When you open Devices window, look at the left menu, it should have options to add audio inputs to devices.

If you don't see that, then click on 'In' section in Devices window. Also, those 'devices' are actually patches that route internal signals to hardware ones, so you can open existing or added input or output in patch panel and see what actual hardware I/O is used there and reroute as desired, if needed.

x.iso · 2018-10-27 04:05:09

So I've basically done most things I wanted with this synth with what I have. But I've been hesitant on releasing this as add-on for few reasons, it's quite CPU intensive (I think without hardware acceleration on it's not even going to work on some machines) and it seems to have some bugs for which I can't figure out the cause yet. So I'm putting it here first kinda as beta test, maybe someone will have ideas on how to optimize it or find what's not working right. Here it is, VidSynth:

it can use either external video signal or internal video player (you can use still pictures as well in it), there's a switch for input in UI. I haven't figured out why drag-n-drop doesn't work through separate list object, but there's a button to show internal list for the player.
P.S.: this patch depends on quite a few different oli_lab modules! Without his modules, it would be quite a pain to make.

VidSynth

Last edited by x.iso (2018-10-27 04:07:43)

x.iso · 2018-10-28 01:30:53

I've reuploaded the patch since I found some mistakes here and there.

Sensomusic

#1 2018-09-28 22:21:40

#2 2018-09-29 09:53:59

#3 2018-09-29 10:09:26

#4 2018-09-29 22:28:49

#5 2018-09-29 22:50:33

#6 2018-09-29 23:28:28

#7 2018-09-30 23:42:38

#8 2018-09-30 23:45:59

#9 2018-10-01 02:27:53

#10 2018-10-02 09:53:04

#11 2018-10-02 18:54:57

#12 2018-10-03 01:17:36

#13 2018-10-03 08:17:16

#14 2018-10-03 14:37:47

#15 2018-10-03 16:07:06

#16 2018-10-03 18:12:46

#17 2018-10-05 22:34:15

#18 2018-10-06 06:30:09

#19 2018-10-27 04:05:09

#20 2018-10-28 01:30:53