StippleDemos Camera widget

I have turned the WebCam Demo into a camera widget. It's not yet very fast and can certainly be optimised, perhaps by using VideoIO.jl, that's why I currently leave it in broken.

Whoever is interested, please try out and comment here.

Jan 13 '22 09:01 hhaensel

You have to add Electron.jl to your packages in order to use this app.

Maybe you have to adapt the camera name according to your system, it's not yet very comfortable.

The widget can be moved by dragging the picture.

Jan 13 '22 09:01 hhaensel

@hhaensel looks like in my case can't access camera. I'm on Macbook. Do I need to allow some sort of camera permission? What OS are you on?

 @ Genie.AppServer ~/.julia/packages/Genie/Qtv4L/src/AppServer.jl:274
┌ Error: UndefRefError: access to undefined reference
│ Stacktrace:
│   [1] getproperty
│     @ ./Base.jl:33 [inlined]
│   [2] getindex
│     @ ./refvalue.jl:56 [inlined]
│   [3] (::var"#16#17")()
│     @ Main ~/Desktop/WebCam.jl:170
│   [4] match_routes(req::HTTP.Messages.Request, res::HTTP.Messages.Response, params::Genie.Router.Params{Any})
│     @ Genie.Router ~/.julia/packages/Genie/Qtv4L/src/Router.jl:470
│   [5] route_request(req::HTTP.Messages.Request, res::HTTP.Messages.Response)
│     @ Genie.Router ~/.julia/packages/Genie/Qtv4L/src/Router.jl:149
│   [6] handle_request(req::HTTP.Messages.Request, res::HTTP.Messages.Response)
│     @ Genie.AppServer ~/.julia/packages/Genie/Qtv4L/src/AppServer.jl:235
│   [7] #24
│     @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:127 [inlined]
│   [8] run_work_thunk(thunk::Genie.AppServer.var"#24#25"{HTTP.Messages.Request, HTTP.Messages.Response}, print_error::Bool)
│     @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
│   [9] remotecall_fetch(::Function, ::Distributed.LocalProcess; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
│     @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:379
│  [10] remotecall_fetch(::Function, ::Distributed.LocalProcess)
│     @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:379
│  [11] remotecall_fetch(::Function, ::Int64; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
│     @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:421
│  [12] remotecall_fetch(::Function, ::Int64)
│     @ Distributed /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:421
│  [13] setup_http_listener(req::HTTP.Messages.Request, res::HTTP.Messages.Response)
│     @ Genie.AppServer ~/.julia/packages/Genie/Qtv4L/src/AppServer.jl:262
│  [14] setup_http_listener
│     @ ~/.julia/packages/Genie/Qtv4L/src/AppServer.jl:261 [inlined]
│  [15] handle
│     @ ~/.julia/packages/HTTP/aTjcj/src/Handlers.jl:254 [inlined]
│  [16] handle(::HTTP.Handlers.RequestHandlerFunction{typeof(Genie.AppServer.setup_http_listener)}, ::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}})
│     @ HTTP.Handlers ~/.julia/packages/HTTP/aTjcj/src/Handlers.jl:277
│  [17] setup_http_streamer(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}})
│     @ Genie.AppServer ~/.julia/packages/Genie/Qtv4L/src/AppServer.jl:251
│  [18] (::Genie.AppServer.var"#7#14"{Int64})(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}})
│     @ Genie.AppServer ~/.julia/packages/Genie/Qtv4L/src/AppServer.jl:117
│  [19] macro expansion
│     @ ~/.julia/packages/HTTP/aTjcj/src/Servers.jl:415 [inlined]
│  [20] (::HTTP.Servers.var"#13#14"{Genie.AppServer.var"#7#14"{Int64}, HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, HTTP.Servers.Server{Nothing, Sockets.TCPServer}, HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}})()
│     @ HTTP.Servers ./task.jl:411
│
└ @ Genie.AppServer ~/.julia/packages/Genie/Qtv4L/src/AppServer.jl:274
 ~/Desktop

Jan 13 '22 14:01 AbhimanyuAryan

I'm on windows 10. You can try to use VideoIO.jl to find out the name of your camera device:

using VideoIO
VideoIO.CAMERA_DEVICES

Then put the name of your camera in place of /dev/video0.

Good luck!

Jan 13 '22 20:01 hhaensel

P.S.: I'm struggling to get VideoIO running on my windows PC.

Jan 13 '22 20:01 hhaensel

I'm on windows 10. You can try to use VideoIO.jl to find out the name of your camera device:
using VideoIO
VideoIO.CAMERA_DEVICES
Then put the name of your camera in place of /dev/video0.

Good luck!

Will look into it. Thanks 🙃

Jan 13 '22 20:01 AbhimanyuAryan

Just found out how to get VideoIO working:

import Base.unsafe_convert
unsafe_convert(::Type{Ptr{Ptr{T}}}, ad::VideoIO.AVDict) where T = unsafe_convert(Ptr{Ptr{T}}, getfield(ad, :ref_ptr_dict))

So there is some probability that we will switch to VideoIO soon.

Jan 13 '22 20:01 hhaensel

Will look into it. Thanks 🙃

You could try

const FMT = "avfoundation"

or

const FMT = "qtkit"

The next version will automate this process ...

Jan 14 '22 11:01 hhaensel

Just submitted a correction for webcam that should make it functional on all platforms. Another version based on VideoIO is released as webcam2.jl. It's slower, as it does the conversion to png in Julia whereas the original version uses ffmpeg conversion.

@essenciary , @AbhimanyuAryan Please test. If both are functional on your machines, we might move it to the working demos.

Jan 14 '22 18:01 hhaensel

@hhaensel webcam2.jl works. Asked for facetime permission but it's very slow. Infact it only took first frame and didn't update new frames. Toggle off turns off the camera.

Jan 15 '22 06:01 AbhimanyuAryan

You have to decrease the FPS_CLIENT in this case. Webcam.jl should also work for you now. Please also decrease FPS_CLIENTthere. Yesterday I tried it it on a slower machine and it only worked with FPS_CLIENT=10

Jan 15 '22 07:01 hhaensel

Toggle turns off the camera so that other programs it like zoom or teams can use it

Jan 15 '22 07:01 hhaensel

will check :)

Jan 17 '22 04:01 AbhimanyuAryan

Please check WebCam3.jl It now supports two updatemodes: webchannels and url, implements cameras in a cleaner way and supports multiple cameras. WebChannel updatemode doesn't populate the log and is more secure.

Jan 17 '22 06:01 hhaensel

yes @hhaensel checking ...

Jan 18 '22 16:01 AbhimanyuAryan

Has anyone tested? Can we move it to Advanced?

Jan 21 '22 12:01 hhaensel

I haven't had time. I'll test tomorrow :)

Jan 21 '22 12:01 AbhimanyuAryan

The latest feature of WebCam3.jl is autosizing of the widget if you drag the side handles. The edge handles don't work properly, but that's not a big issue as they are anyhow difficult to find. For debugging or easier finding of the window handles I have introduced a backgound color on mouse hover. It's set to transparent by default.

Jan 21 '22 13:01 hhaensel

I'm running on macbook m1(16gb ram)

change FPS client but image is still stuck in electron widget for web2.jl
web.jl: [AVFoundation indev @ 0x7f884b705980] Video device not found video=FaceTime HD Camera: Input/output error crashes
webcam3.jl

 paused, 0 bytes waiting), Process(`/Users/abhi/.julia/artifacts/3ba739a4ed073587d96415faecef10a5ccea63ba/Julia.app/Contents/MacOS/Julia /Users/abhi/.julia/packages/Electron/ILvWR/src/main.js /var/folders/0j/d0xd373d2tb7zdzcsqrk3mpw0000gn/T/jlel-a25622e87b6f11ec224cb3815cd79e21 /var/folders/0j/d0xd373d2tb7zdzcsqrk3mpw0000gn/T/jlel-sn-a2562f367b6f11ec0e38d7a7073e8830 s049w1CM47lShNmbrUMDFtoDMbobaHcsdiDdoiY599qzUJ5qRaOFMDaDwvt0YQAZlKFu26h89FRtTlshIsxIYtVnKTNkgJOgNAZpplVILStLgyZK/BHDZPRce2M5dLE2YWPlQ/2bbqnfiV+oWosMVWXrxQdHO8XdJ3WwTrFejA4=`, ProcessRunning), [1 window]), 1, true, Channel{Any}(128))

julia> Received > 3 system signals, hard exiting
[ Info: starting camera with 'video=FaceTime HD Camera'
[AVFoundation indev @ 0x7f855af05980] Video device not found
video=FaceTime HD Camera: Input/output error

Jan 22 '22 10:01 AbhimanyuAryan

So we have to solve two problems:

get VideoIO running on your m1. Did you try running the VideoIO demo withh makie which is on the VideoIO.jl page?
find the root cause of the crash of ffmpeg (webcam3.jl). Maybe it's related to wrong values of sx, sy or fps? If you can use VideoIO, you might be able to find out these values by examining the picture it grabbed. Do you have a new m1 with the 1080p facetime camera or do you have 720p?

Jan 23 '22 00:01 hhaensel

Some weird issue with VideoIO examples:

julia> include("videomakie.jl")
ERROR: LoadError: MethodError: no method matching unsafe_convert(::Type{Ptr{Ptr{VideoIO.libffmpeg.AVDictionary}}}, ::VideoIO.AVDict)
Closest candidates are:
  unsafe_convert(::Type{Ptr{T}}, ::StaticArrays.MArray{S, T, N, L} where {N, L}) where {S, T} at /Users/abhi/.julia/packages/StaticArrays/A0XaR/src/MArray.jl:109
  unsafe_convert(::Type{Ptr{T}}, ::FFTW.FakeArray{T, N} where N) where T at /Users/abhi/.julia/packages/FFTW/pHa9y/src/fft.jl:121
  unsafe_convert(::Type{Ptr{T}}, ::Ptr{Tuple{Vararg{T, N}}}) where {N, T} at refpointer.jl:176
  ...
Stacktrace:
 [1] avformat_open_input(ps::VideoIO.NestedCStruct{VideoIO.libffmpeg.AVFormatContext}, url::String, fmt::Ptr{VideoIO.libffmpeg.AVInputFormat}, options::VideoIO.AVDict)
   @ VideoIO.libffmpeg ~/.julia/packages/VideoIO/EsW3Z/lib/libffmpeg.jl:8571
 [2] open_avinput(avin::VideoIO.AVInput{String}, source::String, input_format::Ptr{VideoIO.libffmpeg.AVInputFormat}, options::VideoIO.AVDict)
   @ VideoIO ~/.julia/packages/VideoIO/EsW3Z/src/avio.jl:195
 [3] VideoIO.AVInput(source::String, input_format::Ptr{VideoIO.libffmpeg.AVInputFormat}, options::VideoIO.AVDict; avio_ctx_buffer_size::Int64)
   @ VideoIO ~/.julia/packages/VideoIO/EsW3Z/src/avio.jl:218
 [4] VideoIO.AVInput(source::String, input_format::Ptr{VideoIO.libffmpeg.AVInputFormat}, options::VideoIO.AVDict)
   @ VideoIO ~/.julia/packages/VideoIO/EsW3Z/src/avio.jl:207
 [5] opencamera(::String, ::Ptr{VideoIO.libffmpeg.AVInputFormat}, ::VideoIO.AVDict; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ VideoIO ~/.julia/packages/VideoIO/EsW3Z/src/avio.jl:1016
 [6] opencamera(::String, ::Ptr{VideoIO.libffmpeg.AVInputFormat}, ::VideoIO.AVDict) (repeats 2 times)
   @ VideoIO ~/.julia/packages/VideoIO/EsW3Z/src/avio.jl:1016
 [7] top-level scope
   @ ~/Git/VideoIO/videomakie.jl:4
 [8] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [9] top-level scope
   @ REPL[9]:1
in expression starting at /Users/abhi/Git/VideoIO/videomakie.jl:4

followed this for solution. No luck: https://discourse.julialang.org/t/running-external-programs-in-windows/72353/6?u=abhimanyuaryan

Need to dig deeper

I have 720p m1 air

Jan 23 '22 02:01 AbhimanyuAryan

Try to execute the two lines that are also in the webcam2 and webcam3. Before you call open camera.

import VideoIO

import Base.cconvert
Base.cconvert(::Type{Ptr{Ptr{VideoIO.AVDictionary}}}, d::VideoIO.AVDict) = d.ref_ptr_dict

cam = opencamera()

Jan 23 '22 07:01 hhaensel

will check/debug on weekend :)

Jan 27 '22 16:01 AbhimanyuAryan

Not very clear what's causing this: https://github.com/JuliaIO/VideoIO.jl/issues/343 says it's the frame rate. Have you tried running this default example?

import VideoIO, GLMakie
import Base.cconvert
Base.cconvert(::Type{Ptr{Ptr{VideoIO.AVDictionary}}}, d::VideoIO.AVDict) = d.ref_ptr_dict

cam = VideoIO.opencamera()

img = read(cam)
scene = GLMakie.Scene(resolution = size(img'))
makieimg = GLMakie.image!(scene, img)
GLMakie.rotate!(scene, -0.5pi)
display(scene)

while isopen(scene)
    read!(cam, img)
    makieimg.image = img
    sleep(1/VideoIO.framerate(cam))
end

close(cam)

ERROR: LoadError: Could not scale frame
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] exec!
   @ ~/.julia/dev/VideoIO/src/frame_graph.jl:109 [inlined]
 [3] execute_graph!
   @ ~/.julia/dev/VideoIO/src/avio.jl:456 [inlined]
 [4] _retrieve!(r::VideoIO.VideoReader{true, VideoIO.SwsTransform, String}, buf::PermutedDimsArray{ColorTypes.RGB{FixedPointNumbers.N0f8}, 2, (2, 1), (2, 1), Matrix{ColorTypes.RGB{FixedPointNumbers.N0f8}}})
   @ VideoIO ~/.julia/dev/VideoIO/src/avio.jl:467
 [5] retrieve!
   @ ~/.julia/dev/VideoIO/src/avio.jl:478 [inlined]
 [6] read!(r::VideoIO.VideoReader{true, VideoIO.SwsTransform, String}, buf::PermutedDimsArray{ColorTypes.RGB{FixedPointNumbers.N0f8}, 2, (2, 1), (2, 1), Matrix{ColorTypes.RGB{FixedPointNumbers.N0f8}}})
   @ VideoIO ~/.julia/dev/VideoIO/src/avio.jl:637
 [7] top-level scope
   @ ~/Git/VideoIO/videomakie.jl:14
 [8] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [9] top-level scope
   @ REPL[1]:1
in expression starting at /Users/abhi/Git/VideoIO/videomakie.jl:13

Jan 31 '22 08:01 AbhimanyuAryan