Dockerized ld-decode (rev6)

Easy way to run ld-decode (rev6) using docker:

docker pull mpownby/ld-decode:rev6

Launching with Ubuntu (with X windows shared, allows 'ld-analyse' to be run)

docker run --rm -v $HOME:/shared -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY -e "QT_X11_NO_MITSHM=1" -ti mpownby/ld-decode:rev6 /bin/bash

Launching with Windows (or other host where you only want to use the command-line tools)

docker run --rm -v /c/your/host/dir/here:/shared -e DISPLAY -e "QT_X11_NO_MITSHM=1" -ti mpownby/ld-decode:rev6 /bin/bash

In the above example, c:\your\host\dir\here is written as /c/your/host/dir/here .  Modify this to point to a folder that you want the docker container to have access to (ie the place where your domesday capture lives).  From within the container, this can be accessed by going to '/shared'.

Caveat: I couldn't get the volume mapping to work with an external USB hard drive.  But it worked with my internal SATA hard drives.

Caveat: Beware that Docker advises that this may cause performance problems but it seems to be working fine for me in my tests.

Once the container starts, run these commands:

• cd ld-decode
• (optional) cat /usage.txt
• python3 ld-decode.py /shared/your_domesday_file_here.ldf /shared/output

This will begin decoding the .ldf file (compressed domesday capture) from the beginning.

NOTE: 'output' is the prefix of the output files that will be created.  You can use a different name but my instructions will assume you are using 'output'.

IMPORTANT: Most domesday captures were recorded while the laserdisc player is spinning up so the beginning of the capture will be "throw away".   You need to let the decode run until you see the CAV frames go back to frame 1.  Here is an example of how this looks:

(stuff before this removed for brevity)

file frame 626 CAV frame 244

file frame 627 CAV frame 245

file frame 628 CAV frame 246

file frame 629 CAV frame 247

file frame 630 CAV frame 248

file frame 631 CAV frame 249

file frame 632 CAV frame 250

file frame 633 CAV frame 251

file frame 634 CAV frame 252

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 635 CAV frame 252

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 636 CAV frame 252

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 637 CAV frame 202

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 638 CAV frame 152

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 639 CAV frame 102

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 640 CAV frame 52

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 641 CAV frame 2

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 642 CAV frame 1

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

WARNING:root:NTSC field phaseID sequence mismatch (player may be paused)

file frame 643 unknown

file frame 644 unknown

file frame 645 CAV frame 1

file frame 646 CAV frame 2

file frame 647 CAV frame 3

file frame 648 CAV frame 4

file frame 649 CAV frame 5

Once you get this far, hit control-c (sometimes twice) to abort the process, take note of the file frame (645 in this example).  Then re-run ld-decode using the following command line:

python3 ld-decode.py /shared/your_domesday_file_here.ldf /shared/output -s 645 --noEFM

Instead of 645, put the appropriate file frame.  Including the --noEFM option may speed up the decoding (which is inherently quite slow even on modern hardware) as no arcade game laserdisc of which I am aware of has any EFM data.

You should see CAV frame 1 show up first in the log file:

file frame 645 CAV frame 1

If you don't, adjust the starting file frame accordingly and try again. For example, try subtracting 1 from the staerting file frame.

This will eventually produce files called output.tbc (which will be huge) and output.pcm.  Follow instructions at this page for further processing.  I recommend using the "(Mostly) Lossless export" instructions in the NTSC wiki section and then doing further processing upon that resulting file.

How to Dockerize ld-decode (rev5)

If you are interested in the Domesday Duplicator, you may have come across the problem of installing the ld-decode software.  This is much easier using Docker because you can just use the Dockerfile that I already created.

Step 1:

Save the following text to a file called 'Dockerfile', in an empty subdirectory.

# known to work with 18.04, other versions may have incompatibilities
FROM ubuntu:18.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && \
    apt-get -y -qq install apt-utils sudo && \
        apt-get -y -qq install locales && locale-gen en_US.UTF-8 && \
    apt-get -y -qq install git \
        libopencv-dev \
        libfann-dev \
        python3-pip \
        python3-tk \
        clang \
        imagemagick wget

# more schlop to install
RUN apt-get -y -qq install libqt5charts5 libqt5charts5-dev qt5-default unzip

# create a normal user so we're not running as root
RUN export uid=1000 gid=1000 && \
    mkdir -p /home/developer && \
    echo "developer:x:${uid}:${gid}:Developer,,,:/home/developer:/bin/bash" >> /etc/passwd && \
    echo "developer:x:${uid}:" >> /etc/group && \
    echo "developer ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/developer && \
    chmod 0440 /etc/sudoers.d/developer && \
    chown ${uid}:${gid} -R /home/developer

# switch to user so it installs from the user's context
USER developer
ENV HOME /home/developer

# python3 schlop
RUN pip3 install numpy pandas scipy matplotlib

# set user's home directory as current directory because that's where we'll do work
WORKDIR /home/developer

# clone and build the rev5 commit
RUN wget https://github.com/happycube/ld-decode/archive/8e8b9f2c18136a1bcdbd3d659d72e42e23135771.zip
RUN unzip 8e8b9f2c18136a1bcdbd3d659d72e42e23135771.zip
RUN mv ld-decode-8e8b9f2c18136a1bcdbd3d659d72e42e23135771 ld-decode
RUN rm 8e8b9f2c18136a1bcdbd3d659d72e42e23135771.zip

RUN cd ld-decode && make && cd tools && qmake && make all && sudo make install

Step 2:

Build docker image using this command:

docker build -t lddecode_rev5 -f Dockerfile .

Step 3:

Run docker image using this command:

docker run --rm -v $HOME:/shared -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY -e "QT_X11_NO_MITSHM=1" -ti lddecode_rev5 /bin/bash

From inside the image, the /shared folder will point to your native host's home directory.

Follow instructions here for usage.  Also, check out the "ld-analyse" UI app, it is very slick.

The nice thing about using the docker container is that you don't have to worry about installing a bunch of ld-decode dependencies to your native host operating system.  To uninstall, you simply remove the docker image.  Also, you can run more than one container in parallel if desired.

How to dockerize Intel Quartus 18.01

I've been working with the Domesday Duplicator lately and one of the pieces of hardware that they use is the DE0-Nano board.  Domesday documentation focuses on linux tools so I booted into my Ubuntu and got the DE0-Nano IDE software (called 'Quartus') working.  While I was successful, I found it a bit of a pain to figure out which components of the software to download, and would've preferred a pre-built VM or a Docker image.  The Quartus license likely does not allow distribution of pre-built Docker images, so I have outlined the steps to create one from scratch here.

The benefits to running this program from a docker container are:

• The docker image, once created, will likely work on many more platforms than what the native installation option offers.
• Building and running the Docker image is relatively easy.
• Installing and uninstalling is simple and clear.

So here are my instructions to Docker-ize Quartus v18.01:

• Be using some linux flavor (I am on Ubuntu 18.04).
• Install Docker and configure it so that normal users can access it (beyond scope of this blog post).
• Create a file called /etc/udev/rules.d/51-usbblater.rules which has the following contents: (this file is necessary to allow non-root access to the USB device)
  
  SUBSYSTEM=="usb", ATTRS{idVendor}=="09fb", ATTRS{idProduct}=="6001", MODE="0666"
  
• Run "sudo udevadm control --reload-rules" to make the change take effect
• Download Quartus 18.1 Lite for linux, including the Cyclone 4 package.  The filenames I downloaded are called QuartusLiteSetup-18.1.0.625-linux.run and cyclone-18.1.0.625.qdz .
• Create a new subdirectory and copy QuartusLiteSetup-18.1.0.625-linux.run and cyclone-18.1.0.625.qdz into it.
• Inside this subdirectory, create a file called Dockerfile and paste this into it:
  
   # ubuntu 18.04 is officially supported by this version of quartus
  # libpng12 included in 16.04 but not in 18.04, so we use 16.04 for convenience
  FROM ubuntu:16.04
  
  ENV DEBIAN_FRONTEND noninteractive
  
  ARG QUARTUS=QuartusLiteSetup-18.1.0.625-linux.run
  ARG CYCLONE=cyclone-18.1.0.625.qdz
  
  COPY $QUARTUS /$QUARTUS
  COPY $CYCLONE /$CYCLONE
  
  RUN apt-get update && \
      apt-get -y -qq install apt-utils sudo && \
          apt-get -y -qq install locales && locale-gen en_US.UTF-8 && \
      apt-get -y -qq install software-properties-common \
                             libglib2.0-0:amd64 \
                             libfreetype6:amd64 \
                             libsm6:amd64 \
                             libxrender1:amd64 \
                             libfontconfig1:amd64 \
                             libxext6:amd64 \
                                  libpng12-0:amd64 \
                             xterm:amd64 && \
      chmod 755 /$QUARTUS
  
  # create a normal user so we're not running as root
  RUN export uid=1000 gid=1000 && \
      mkdir -p /home/developer && \
      echo "developer:x:${uid}:${gid}:Developer,,,:/home/developer:/bin/bash" >> /etc/passwd && \
      echo "developer:x:${uid}:" >> /etc/group && \
      echo "developer ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/developer && \
      chmod 0440 /etc/sudoers.d/developer && \
      chown ${uid}:${gid} -R /home/developer
  
  # switch to user so it installs from the user's context
  USER developer
  ENV HOME /home/developer
  # install quartus as the user (not root)
  RUN    /$QUARTUS --mode unattended --unattendedmodeui none --installdir /home/developer/altera_lite --accept_eula 1 && \
      sudo rm -f /$QUARTUS && \
      sudo rm -f /$CYCLONE
  
  # run from xterm to capture any stdio logging (not sure there is any, but can't hurt)
  CMD xterm -e "/home/developer/altera_lite/quartus/bin/quartus --64bit"
• From within this subdirectory (which contains the three files), run this command:
  "docker build -t quartus18 ."
• It will take ~15 mins depending on the speed of your system.  When it finishes, your Docker image will be generated and will be called "quartus18".  You may now optionally delete the sub-directory that you created to generate the docker image.
• Plug in your DE0-Nano to a USB port.  Be sure to unplug any other sources of power (for example, the FX3 board, if you have a DdD 'stack' ) or you will get unexplained errors.
• From a desktop session, run the following command:
  
  docker run --rm -v /sys:/sys:ro -v $HOME:/shared -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY -e "QT_X11_NO_MITSHM=1" --privileged -v /dev/bus/usb:/dev/bus/usb quartus18
  
• If all goes well, the Quartus app will launch inside of the container.  Your host's home folder will be accessible as /shared in the root.  Follow the instructions from DdD site.  You may also find this youtube video helpful.
• When you exit the program, the container will automatically delete/uninstall itself.

The only thing left on your host system once you exit the container will be the file you put inside of /etc/udev/rules.d and the docker image that you created.  Uninstalling is as easy as removing this file and deleting the docker image.  Simply re-execute the "docker run" command above in the future to reprogram.

How to dockerize Cypress FX3 Linux USB programmer

I've been working with the Domesday Duplicator lately and one of the pieces of hardware that they use is the FX3 board from Cypress.  Domesday documentation focuses on linux tools so I booted into my Ubuntu and got the FX3 USB programmer working.  It wasn't that easy, though.  For example, I had to modify the source code to fix a build error.  Installation seems to be one-way, meaning there is no uninstall (which seems somewhat rude).  It would've been cool to run this in a VM so that when I was done, I could just shut the VM down.  That is kind of a huge pain, though.  A better option is Docker!  The problem is, the software's license likely prohibits it being distributed as a docker container, so one must create one from scratch.  Fortunately, after some experimenting, I got it all working.

The benefits to running this program from a docker container are:

• The docker image, once created, will likely work on many more platforms than what the native installation option offers.
• You don't have to install all of the qt4 dev packages on your native OS.
• Uninstalling is simple and clear. (no weird files leftover in weird places)

So here are my instructions to Docker-ize Cypress's FX3 linux USB programmer:

• Be using some linux flavor (I am on Ubuntu 18.04).
• Install Docker and configure it so that normal users can access it (beyond scope of this blog post).
• Create a file called /etc/udev/rules.d/88-cyusb.rules which has the following contents: (this file is necessary to get flash programming mode to work)
  
  # Cypress USB driver for FX2 and FX3 (C) Cypress Semiconductor Corporation / ATR-LABS
  # Rules written by V. Radhakrishnan ( rk@atr-labs.com )
  # Cypress USB vendor ID = 0x04b4
  KERNEL=="*", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ACTION=="add", ATTR{idVendor}=="04b4", MODE="666", TAG="cyusb_dev", RUN+="/usr/local/bin/cy_renumerate.sh A"
  KERNEL=="*", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ACTION=="remove", TAG=="cyusb_dev", RUN+="/usr/local/bin/cy_renumerate.sh R"
• Run "sudo udevadm control --reload-rules" to make the change take effect
• Create a script called /usr/local/bin/cy_renumerate.sh that contains this:
  
  #!/bin/bash
  pid=`pidof cyusb_linux`
  
  if [ "$pid" ]; then
      kill -s SIGUSR1 $pid
  fi
• Type "chmod a+x /usr/local/bin/cy_renumerate.sh" so the script can be executed
• Download the Cypress FX3 linux SDK (try here or search for "cypress fx3 linux sdk")
• Extract the archive and locate the file called cyusb_linux_1.0.5.tar.gz .
• Create a new subdirectory and copy cyusb_linux_1.0.5.tar.gz into it.
• Inside this subdirectory, create a file called Dockerfile and paste this into it:
  
  # this is known to work on 18.04, may not work on later versions
  FROM ubuntu:18.04
  
  ARG DEBIAN_FRONTEND=noninteractive
  ARG CYUSB=cyusb_linux_1.0.5.tar.gz
  
  COPY $CYUSB /$CYUSB
  
  RUN apt-get update && \
          apt-get -y -qq install apt-utils && \
      apt-get -y -qq install sudo qt4-default \
          less make g++ dialog libusb-1.0-0-dev \
          xterm
  
  RUN export uid=1000 gid=1000 && \
      mkdir -p /home/developer && \
      echo "developer:x:${uid}:${gid}:Developer,,,:/home/developer:/bin/bash" >> /etc/passwd && \
      echo "developer:x:${uid}:" >> /etc/group && \
      echo "developer ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/developer && \
      chmod 0440 /etc/sudoers.d/developer && \
      chown ${uid}:${gid} -R /home/developer
  
  # switch to user so it installs from the user's context
  USER developer
  ENV HOME /home/developer
  
  # needed so cypress app can program itself to go into flash mode
  ENV CYUSB_ROOT /home/developer/cyusb_linux_1.0.5
  
  # build/install
  RUN    sudo chmod a+r /$CYUSB && \
          tar xvfz /$CYUSB -C /home/developer && \
          sudo rm -f /$CYUSB && \
          cd $HOME && \
          cd cyusb* && \
          make && \
          sed -i 's/errno/_errno/g' gui_src/main.cpp && \
          sudo ./install.sh
  
  # put /usr/local/lib in the lib search path so running 'cyusb_linux' from anywhere works
  RUN     sudo sh -c "echo '/usr/local/lib' >> /etc/ld.so.conf.d/cyusb.conf" && \
          sudo ldconfig
  
  # launches in xterm so we can see log messages
  CMD xterm -e "cyusb_linux"
• From within this subdirectory (which contains the .tar.gz file and Dockerfile), run this command:
  "docker build -t cyusb ."
• It will take ~5 mins depending on the speed of your system.  When it finishes, your Docker image will be generatedand will be called "cyusb".
• Plug in your FX3 to a USB port.
• Make sure that the FX3 image that you want to program to the FX3 is stored somewhere under your home directory.
• Run the following command:
  docker run --rm \
      -v $HOME:/shared:ro \
      -v /tmp/.X11-unix:/tmp/.X11-unix \
      -e DISPLAY \
      -e "QT_X11_NO_MITSHM=1" \
      --privileged \
      -v /dev/bus/usb:/dev/bus/usb \
      cyusb
• If all goes well, the cyusb app will launch inside of the container.  Your host's home folder will be accessible as /shared in the root.  Follow the instructions from DdD to complete the programming.
• When you exit the program, the container will automatically delete/uninstall itself.

The only thing left on your host system once you exit the container will be the file you put inside of /etc/udev/rules.d, the little script in /usr/local/bin, and the docker image that you created.  Uninstalling is as easy as removing those two files and deleting the docker image.  Simply re-execute the "docker run" command above in the future to reprogram.

Neat, eh?

Docker tips

I've had the need to learn more about Docker lately and had some trouble getting stuff working.  Here are some helpful links that I used to solve problems, put here so I can refer to them later (or so anyone else can benefit from them).

To docker-ize UI applications:

http://wiki.ros.org/docker/Tutorials/GUI
https://blog.docker.com/2013/07/docker-desktop-your-desktop-over-ssh-running-inside-of-a-docker-container/
http://fabiorehm.com/blog/2014/09/11/running-gui-apps-with-docker/

To expose USB devices to docker containers:

https://stackoverflow.com/questions/24225647/docker-a-way-to-give-access-to-a-host-usb-or-serial-device