semaphore - PDF

Document Sample
semaphore - PDF Powered By Docstoc
					                    Decoding the San Jose Semaphore

            Mark Snesrud                                       Bob Mayo
         msnesrud@yahoo.com                              bob-sjs@bobmayo.com


                                       OVERVIEW

The San Jose Semaphore is a piece of animated public art located atop Adobe’s building
in San Jose, beginning operation on August 7, 2006. It consists of four orange
illuminated discs that rotate every 7.2 seconds, along with a soundtrack available in the
immediate proximity on AM 1680 radio. In addition to the esthetics of the art, the
movements of the discs and the sounds convey an encoded message. This report
describes the successful decoding of that message. More information on the San Jose
Semaphore is available at www.sanjosesemaphore.org.




        The Semaphore atop Adobe's headquarters in San Jose, California.




Introduction
It was a Thursday night in downtown San Jose, where we decided to meet for a beer and
dinner along with a few other friends and co-workers. Not much was going on at the
place where we ate dinner, so we decided to explore the surrounding areas to see if we
could locate a fun place to hang out. As we crossed Almaden Boulevard, the orange
discs of the San Jose Semaphore stood out against the night sky, presenting a display that
engaged our curiosity. Not much progress was made on our exploration of San Jose that
night, but two of us decided to try to solve the puzzle that the Semaphore posed.

Scaling the Wall Next to the Elevator
We noticed the online simulcast of the semaphore, but wasn’t sure if contained the
encoded message, too. The simulcast did not behave exactly like the version on the
Adobe building, in that the actual art varied its appearance when airplanes flew over,
while the online simulcast did not. Bob conjectured that we would need to videotape a
few hours of the semaphore, and perhaps even set up a webcam for long term operation.




               Fig. 1: Filming the Semaphore. The Semaphore was easily
                visible throughout the area, although clear reception of the
            associated AM radio signal was problematic. Since 24/7 monthly
            parking rental was available for this location, we thought we might
                      leave a vehicle there with a webcam and uplink.




While it was easy to film the semaphore, capturing a clean audio track from AM 1680
radio was more difficult. Using a simple handheld radio we were not able to get a clean
signal anywhere in the area. Switching to a larger boom-box type of radio helped a little
bit, but was still not good enough if we were counting on it to capture all the data. So
Bob purchased a Grundig G1000A radio, which reviews on the web said was an excellent
radio for receiving weak AM radio stations.
              Fig 2: A Succession of Radios. Several radios were tried in an
             attempt to get a strong AM audio signal. The GE SuperRadio on
                  the right is specially designed to receive weak signals.




Alas, the Grundig worked better in most locations, but still did not produce a signal we
were satisfied with. Unexpectedly, one of the strongest signals was received far away
from the Adobe building, near a railing that we hypothesize had something to do with
enhancing reception. But the signal was still weak and faded in and out more than we
liked. Next step: a GE SuperRadio, the king of long distance AM radio reception.




                                                  Fig 3: Way out here?
                                             Surprisingly, AM radio reception
                                            was good in this location far behind
                                            the Adobe building, perhaps due to
                                             the railing or other metal objects
                                                   acting like antennas.
The GE radio lived up to its claim to fame. When a block away from the semaphore, it
managed to pull in a radio station from Boulder, Colorado instead of the semaphore. But
when close to the semaphore, it locked on and produced an excellent signal. Somehow,
we needed a way to webcast this signal from near the Adobe building. The news rack in
front caught our eye.




                                                             Fig 4: Future home of
                                                            webcam and radio? This
                                                          spot in front of the semaphore
                                                             had excellent AM radio
                                                            reception and also a clear
                                                          view of the semaphore. And,
                                                           conveniently, room for one
                                                          more newspaper vending box
                                                                    to be added.




This news rack is located close to the semaphore and provides excellent AM radio
reception with the GE SuperRadio, and also has a clear view of the Semaphore. We
wondered: could we install our own newspaper box containing a webcam, the
SuperRadio, and a wireless uplink? Wi-Fi access is available, so…..

But maybe all that was unnecessary. After all, we had the Simulcast to work from. And
deriving data from the Simulcast seemed like a much easier proposition. We found
software that could take snapshots of the simulcast at 10 frames per second, and other
software that could process these into a form that clearly could be recognized by image
processing software we intended to write.
                                                          Fig 5: Recognizing the discs.
                                                             It appeared to us that we
                                                          could write image recognition
                                                            software to determine the
                                                           movement of the discs. The
                                                             images to the left show a
                                                           screen captured image along
                                                            with the result of minimal
                                                                 image processing.




Furthermore, we noticed that the audio track from the simulcast was clean, and consisted
of six distinct segments. While it would take a great deal of work, we believed we could
automatically recognize these segments.



                                                               Fig. 6: Recognizing the
                                                              sounds. Each 7.2 second
                                                               period of the soundtrack
                                                          began with a 1.8 second beep
                                                          with sounds of discs spinning,
                                                          another 0.6 second beep, a 1.2
                                                          second recording of a spoken
                                                                letter, and a 0.9 second
                                                            spoken number followed by
                                                           two beeps. The first of these
                                                          was 0.2 seconds long and was
                                                              followed by a 2.5 second
                                                             period consisting of a beep
                                                                  followed by silence.




Clearly, we had our work cut out for us. While we were confident we could use software
to automatically recognize the beeps and words of this audio, it was not something we
would be able to do in an evening. Fortunately Mark had a better solution.
                                                     Fig 7: XML File. After
                                                  discovering that an XML file
                                                   like this was available from
                                                    the Simulcast web site, we
                                                   abandoned efforts to decode
                                                  the actual semaphore and the
                                                     image and sounds of the
                                                    simulcast. Unless the data
                                                     did not match the actual
                                                   semaphore, the file gave us
                                                          all we needed.




“I found an XML file” was the utterance that changed all of this. As it turns out, the
simulcast downloaded an XML file containing the current transmission, and we could
reliably fetch this file directly from the Simulcast web site.

Now we were in business. We would not have to plant a fake newspaper box in front of
Adobe after all, nor would we have to face up to the resulting interrogation from
Homeland Security after they blew up our unexplainable black box with a water cannon.
The first of many wrong turns had come to an end.


The Gang of 41
After acquiring a few hours of data, our next task was to find any sort of pattern. We
noticed that disc position is expressed as the number of eight’s of a circle rotated, with
positive numbers indicating clockwise rotation and negative numbers counterclockwise.
The normal range was -8 to +8. But when we plotted these values for one of the discs,
we saw a chart shown in Figure 8.
              Fig 8: Finding the header. Every so often the discs have a
              period of restricted movement, always in the range +1 to +4
                rather than the usual range -8 to +8. This turned out to
                  indicate an important structure we called a “header”.




Every so often the discs restricted their movement for 41 time periods. This looked
significant to us, so we referred to these sections as headers. A number of attempts were
made to find a pattern in the disc movements, but we could not decode a message. After
other false starts, we looked at sound3 and sound4 in each header (Figure 9).
                   Fig 9: An incrementing number? Most of a header
               appeared constant, and other parts of it appeared to change in
                a way we did not understand. But some places, indicated by
               the red bar, appeared to behave like an incrementing counter.




We noticed that most of the sound3 and sound4 data was always the same in each header,
while other data changed in a way we did not understand. But a small part of the header
had the behavior of an incrementing number: the digits changed in a pattern that repeated
every 10 headers, and after 10 headers the next position to the left changed to its next
pattern. So the natural hypothesis was that this was an incrementing number, and that the
10 patterns saw represented the digits 0 through 9. A character-substitution formula was
developed that mapped the data we saw into these digits and, after some work, also
mapped the other parts of the header into readable ASCII characters. The result is shown
in Figure 10.
                Fig 10: Decoded Headers. After noticing the counter and
               decoding the digits, we were able to apply the decoding to the
               entire header. Each header contained a seven-letter keyword,
                        the purpose of which we did not understand.




Clearly we have successfully decoded the header, showing it contained a counter and a
seven-letter keyword. After some work, we were also able to demonstrate that the discs
and other sounds in the header could be automatically generated from just the keyword,
telling us that there was no more information in the header for us to decode.

So, at this point we had headers which contained seven-letter keywords, each of which
was followed by a block of data. But what did the keywords mean? Were they random?

Some of the keywords were unusual and not ones that would appear often in normal text.
These allowed us to Google for the keywords and discover they were all from the book
“Ulysses”, by James Joyce. They appeared in the Semaphore’s transmission in the same
order as the book, but some words in the book were skipped over. We wondered why.

It turns out that certain words in the book are eliminated from the words used as
keywords by a set of rules. Words followed by apostrophe “s” are excluded. Some other
words were too, but we didn’t know why. For a while we entertained the possibility that
the message was encoded in the pattern of exclusions, but we could not decode anything
from this. Eventually we discovered that words containing the letters H, P, and X are not
included. These happen to be the 8th, 16th and 24th letters of the alphabet.
Hmmm….what does it all mean?


There’s Something in Here
From our experience in generating the entire header from just the keyword, we learned
that one step in the process was to use a function of sound3 and sound4 to produce the
lower two bits of the discs, and then to set the upper bits to zero. On a hunch, we tried
the same function to generate the lower bits of the discs in the data block, too, and found
it worked. We didn’t know how to compute the upper bits, though.

We shifted our attention again to sound3 and sound4. Was there any pattern there? We
decided to compute the percentage of (sound3, sound4) pairs that were used versus the
number of combinations that were possible. We found that a large portion was used, but
not all. However, too many were used to give us much hope that each pair represented a
character as in the header, since in that case we would expect to see 26 combinations plus
a few for punctuation, or perhaps 52 plus punctuation if upper and lower case were used.
Our analyses showed many more than that.

After searching the web for words like “Ulysses cipher”, we discovered that Ulysses S.
Grant (no relation to the book Ulysses, of course) was well known for the cryptographic
work of his team in decoding the confederate cipher. While reading about this, we
learned about poly-alphabetic ciphers and the Vigenere cipher. In that cipher, each letter
in the keyword is used in rotation to encode the next character in the message. In our
case, we knew the keyword for each block and, more importantly, that it was seven letters
long.

We decided to take a large block and sort the pairs of sounds into seven bins in rotation,
with the first pair of the block going into bin 0, the second pair into bin 1, etc. According
to the rotation, the eighth pair went into bin 0, and so on. In this way, each bin would
correspond to one letter of the keyword of a Vigenere cipher. If there was a function that
encoded each character according to the associated keyword letter, in rotation ala the
Vigenere cipher, each bin should contain a small number of pairs, corresponding to the
alphabet (26 or 52 plus punctuation). This was not the case, however. Dead end.

Around this time, it was discovered that the last pair of sounds in the block was special,
in that a simple function would always turn it into an alphabetic character, and
furthermore that character was contained in the block’s keyword. That could not be just
random, and gave us more reason to believe that somehow each pair of (sound3, sound4)
pairs in the block was somehow created by the use of the corresponding keyword
character.

Perhaps the discs and the minor sounds (sound1, sound2, sound5, and sound6) somehow
came into play? In the header they were redundant, but we didn’t know if they were or
not for the data blocks. We decided to do a large number of experiments to see if a
particular set of data (such as sound1) could be used to predict another set of data (such
as bit 2 of disc 3). To do this, we wrote a function that checked for consistency. For
instance, figure 11 shows code that tries to construct a table that takes a value A and
predicts the value of B. However, if it finds a value A that maps into two different values
of B, then we know that A cannot be used to predict B.
          Fig 11: Predicting Data. We weren’t sure if the data in the discs or minor
       sounds was important, so we used a function like this to see if we could compute
                     them from other data, showing they were redundant.




After many false results, we discovered that the sign of disc 2 (which indicates either
clockwise or counterclockwise rotation) was predicted by a combination of sound1 and
sound2. By manual inspection, we discovered the function and found it used the upper
bit of sound1 and both bits of sound2: “sign = a_1 ^ ((a_1 ^ ~b_1) & (a_1 ^
~b_0))”. A similar function predicted the sign of disc3 from sound5 and sound6.

Further work on decoding the sound3-sound4 pairs had not lead anywhere, so we decided
to continue work finding bits in the discs that might be significant. We focused on bit 2
of disc3, and found that the previous values of the discs (from the previous time period)
along with the current sounds could be used to predict it. After much experimentation,
however, could not find the function. We knew that this problem was similar to the logic
synthesis problem found in designing microprocessors, so we decided to apply some of
those techniques.

We downloaded a classic logic optimization tool, called Espresso, to see if it could do
better. To use as input to the tool, our software produced a table showing the value of bit
2 of disc3 for each combination of previous discs and current sounds. We hoped that
Espresso would be able to grind away on this data and present us with a simple formula,
similar perhaps to the one we found for the sign. With only an underpowered laptop
available for this purpose, we had our result in 14 hours.

There was indeed such an equation. Part of it is shown in Figure 12. It was 4,142 pages
long. This is not the sort of equation we would expect the semaphore designer to choose.
Furthermore, its length indicated it probably was more like a restatement of the data seen
so far, rather than a succinct way to compute bit 2 of disc3. To test this hypothesis, we
substituted a random number for bit 2 of disc3 in our data, and re-ran our process. The
result took 36 hours to compute this time, but produced an equation of similar size. We
don’t know what to make of this, but our hypothesis was that bit 2 of disc3 is either
random, produced by a good-quality hash function, or produced by a number of
exclusive-ors (which are not handled well by Espresso). Or else it contained real data.
We couldn’t tell, and this was another dead end of sorts.




           Fig 12: Predicting bit 2 of disc3. This bit is indeed predicted by the
              previous discs and the sounds, but only with an absurdly large
           equation. To show this equation in its entirety it would require 4,142
            pages. We doubt Ben Rubin, the designer of the semaphore, would
                            have written an equation that long.



Numerous other ideas were investigated which led nowhere. No new patterns could be
found in the discs or in sounds 1, 2, 5 or 6. Eventually we turned our focus back to the
(sound3, sound4) pairs.
The Final Leg
Once we had collected over two weeks of the transmission data, it became possible to
perform frequency analysis on the (sound3, sound4) pairs. We made a table of the
number of occurrences of each pair in the data block portion of the transmission. A
couple of things were immediately obvious.

First, the sign of sound4 did not affect the frequency. This led us to ignore the sign in
subsequent frequency analysis. Second, certain pairs were much more frequent than
other pairs. We spent a lot of time trying to discover a pattern which would explain why
some pairs were very frequent and some pairs never occurred. Unfortunately we were
unsuccessful.

A breakthrough happened when we decided to do frequency analysis on each data block
independently. Most of the data blocks were small and therefore hard to obtain any
meaningful information from. There were however a few larger blocks with 2000 or more
lines of data. These blocks provided important clues.

Each data block seemed to have its own 'signature'. Certain pairs were very common in
one block, but were completely absent in another block. We spent time looking for
patterns in the individual blocks, but again that was unsuccessful.

                                                     sound3
                  1    2    3    4    5    6    7    8    9    10   11   12   13   14   15   16
         s   1    55   0    0    5    0    1    0    15   0    0    0    0    20   0    1     0
         o   2    2    0    28   39   0    0    0    39   0    16   5    3    0    5    0     0
             3    53   1    2    20   0    0    0    0    0    3    10   0    4    0    0     0
         u
             4    5    98   29   29   13   0    0    1    17   0    0    0    0    23   51    7
         n
             5    0    0    11   21   0    0    0    13   0    9    2    0    0    0    0     0
         d   6    66   0    38   5    0    38   23   27   0    1    0    7    0    0    26   14
         4   7    0    0    11   3    0    0    14   0    9    0    15   0    0    0    0     0
             8    1    1    5    29   23   5    7    8    2    5    25   0    0    23   0    48
             9    55   0    14   2    4    0    3    0    0    17   0    7    0    0    0     0
             10   3    21   22   41   1    3    0    17   5    0    0    0    19   0    8     0
             11   0    23   9    4    0    0    0    14   0    0    0    0    19   0    0     0
             12   23   8    5    0    21   3    7    0    1    17   17   4    16   21   0    38
             13   11   0    10   0    13   0    0    0    21   0    0    0    0    22   0    19
             14   65   1    5    38   3    0    0    16   0    5    9    0    0    10   0    20
             15   58   0    16   0    0    7    11   0    0    22   0    0    0    4    2     0
             16   21   32   26   0    5    4    1    0    23   22   22   21   30   0    7     0


         Fig 13: Signature for keyword FACTION. This table shows the number of
  occurrences of each (sound3, sound4) combination in data block following the header
 with keyword FACTION. Different keywords have tables that may look very different.
  For example, the table for keyword BUILDER has 56 occurrences of (2, 2) compared
   with 0 for FACTION, and 0 occurrences of (1, 3) compared with 53 for FACTION.
We had an idea that the 7-letter keywords in the header were related to the data, and
eventually we were able to confirm it. By comparing two data blocks whose keywords
were similar, we noticed a high correlation of the frequency pairs in each one. For
example, the keywords LEONARD and SNARLED or RUTLAND and ROTUNDA
share 6 out of 7 letters. The frequency pattern of the pairs for these blocks are very
similar.

However, ROTUNDA and WIMBLES share 0 letters, and their frequency patterns are
very different. So we now knew that the letters in the keyword affected the (sound3,
sound4) values. By looking at blocks whose keywords differed by only a single letter,
we tried to identify which pairs seemed to be affected by each letter of the alphabet.
Unfortunately this was not possible since we had not captured enough large data blocks
and the smaller data blocks didn't contain enough pairs to be meaningful.

We finally made a breakthrough when we developed a new way to present the frequency
analysis data. We threw out the smallest data blocks, those with less than 750 entries.
For the medium and large blocks, we normalized the frequency data to make the
frequency of each pair independent of the total number of entries in the data block.

For each (sound3, sound4) pair, we listed the letters in the keyword along with the
normalized frequency of that pair. When we sorted each list based on frequency, some
interesting patterns appeared. All the keywords which contained a certain letter would be
at the top of the list. Then the words which contained another letter would be next on the
list. For example, all the keywords at the top of the list for pair (1, 4) had the letter D in
them. The next keywords in that list all had the letter E. And the keywords which
contained both D and E were listed at the very top.
                                                           Normalized frequency
              Letters in the keyword             Keyword   of pattern (1,4)   _
              DE G         NO      TU            TONGUED    88
          A DE      I      N   R T               ARDENTI    87
          A CDE              O R T               REDCOAT    85
          A DE             N       TUV           VAUNTED    84
              DE G         N   R TU              GRUNTED    79
          AB DE G          NO                    BONDAGE    75
             CDE    I      N   RS                DISCERN    74
          A DE          L N    RS                SNARLED    71
            B DE    I L        R U               BUILDER    70
             CDE    I        O R       V         DIVORCE    69
          A CDE         L O R                    ORACLED    69
          A D       I L        RS            Z   LIZARDS    60
          A D       I LMN            U           MAUDLIN    59
              D     I    MNO     S U             DOMINUS    58
          AB D G I         N             W       WINDBAG    53
          AB E      I L          ST              BASTILE    25
          A     E   I      N Q TU                ANTIQUE    24
                E   I LM O RS                    LISMORE    20
          A     E        M     RST         Y     STREAMY    20
          A     E     J L N          UV          JUVENAL    19
                E   I K N      RS        W       WINKERS    18
            B E G       L NO     S               BELONGS    18
             C E    I      NO    ST              NOTICES    17
                E   IJKL       R           Y     JERKILY    17
            B E     I K        RST               BRISKET    17
          A C E         LM         TU            CALUMET    15
            B EF I L           R           Y     BRIEFLY    15
          A     E     J MNO      S               NOSEJAM    15
            B E     I L O Q          U           OBLIQUE    15
          A     E G          O   S V Y           VOYAGES    14
             C E    I KL N     R                 CLINKER    14
            B E     I KL       R U               BULKIER    10
          ABC     G          O   ST              COSTBAG     6
          A C F I          NO      T             FACTION     5
          A             L O QRS U                SQUALOR     2
                  G I      NO RS U               ROUSING     0

       Fig 14: Frequency Table for pattern (1, 4). The table for pattern (1, 4) shows
       that keywords with a D in them have the most occurrences of this pattern,
       followed by keywords with an E. Keywords with both D and E have even more.


We learned which pairs seemed to be a function of which keyword letters, but still didn't
know what it meant. Also, for many of the pairs it wasn't clear which letters really were
at the top of the list. We knew we had to come up with some sort of mapping function to
explain the data. Based on the progress we had made so far, we expected that mapping
function to be fairly straight-forward and elegant.

Eventually we noticed that aside from (1, 1), all of the pairs which seemed to be a
function of the letter A were in two columns, (3, X) or (4, X). After looking at it for a
while we figured that each pair in these columns represented the encoding of a different
letter of the alphabet with the letter A.

We discovered an encoding which was simple and fit the data well, based on our
knowledge of the frequency of certain letters of the alphabet in most English writing. For
instance, the letters E, T, and A occur most often. The mapping we came up with for the
letter A was:
                    (3, 2) = A
                    (3, 3) = B
                    (3, 4) = C
                     ...
                    (3, 16) = O
                    (4, 1) = P
                    (4, 2) = Q
                    ...
                    (4, 10) = Z

We then tried to come up with a mapping for the letter B, but it was difficult since few of
the keywords had the letter B. But eventually we concluded that this was the correct
mapping:
                  (5, 4) = A
                  (5, 6) = B
                  (5, 8) = C
                  ...
                  (5, 16) = G
                  (6, 2) = H
                  (6, 4) = I
                  ...
                  (8, 6) = Z

From there we were able to come up with a simple mapping of the letters of the alphabet
for each of the possible keyword letters. We found that if we assumed that the letters
were ASCII encoded our mapping was also able to handle punctuation and spaces.

                  L = keyword letter (ASCII code)
                  C = character to encode (ASCII code)
                  x = (L-64) * (C-32+1) - 1
                  sound3 = ((x >> 4) & 0xF) + 1
                  abs(sound4) = (x & 0xF) + 1

Using this mapping, we listed the possible ASCII characters for the entries in each data
block based on the 7 letters of the keyword for that block. Even though we still didn't
know which keyword letter to use on each entry, by examining the ASCII characters we
could easily see sentences form in the data blocks.
                                Possible     Possible
                   (Sound3,     decoded      encoding     decoded
        Keyword     sound4)     chars        letters      char _
       FACTION      4,-14:     "T]H"        "FAN"          T
       FACTION      3, 9:      "H@"         "AI"           H
       FACTION      8, 2:      "2E^"        "FCN"          E
       FACTION      2, 4:      "M3 E"       "FATN"
       FACTION      7, -9:     "B&"         "CO"           B
       FACTION     16, 8:      "EWC"        "TIN"          E
       FACTION      8, 8:      "3G%'"       "FCTO"         G
       FACTION      5, -8:     "+7I'W["     "FCTION"       I
       FACTION     11, 7:      "N"          "I"            N
       FACTION     13, -1:     "N"          "O"            N
       FACTION      5,-12:     "VI"         "TN"           I
       FACTION      2,-10:     "N9"         "FA"           N
       FACTION      3, 8:      "G!"         "AT"           G
       FACTION      1, 3:      "" "         "AC"
       FACTION     12, 16:     "?_O_?"      "FCTON"        O
       FACTION      6,-15:     "F"          "I"            F
       FACTION      1, 15:     ".$ "        "ACO"
       FACTION     14, 13:     "T"          "I"            T
       FACTION      4, -4:     "SH5"        "ATN"          H
       FACTION      7, -6:     "0AY"        "FCO"          A
       FACTION      4, -5:     "T"          "A"            T
       FACTION      1, 6:      " %!D"       "FACN"
       FACTION      4, 1:      "P"          "A"            P
       FACTION     15, -4:     "ED"         "FT"           E
       FACTION     13,-10:     "R"          "N"            R
       FACTION      5, 9:      "F"          "O"            F
       FACTION     11, 16:     "[O"         "TI"           O
       FACTION     16,-13:     "R"          "O"            R
       FACTION      9, -4:     "5KL[M"      "FCTON"        M
       FACTION     13,-12:     "A6"         "FT"           A
       FACTION      3,-15:     "N"          "A"            N
       FACTION      7, 12:     "1C>+9"      "FCTIN"        C
       FACTION      3, 6:      "PE4"        "FAN"          E

Fig 15: Manual decoding. This figure shows how the manual decoding of the
first decoded sentence was performed. The (sound3, sound4) pairs from the
transmission were decoded with each of the 7 letters in the keyword. Some of the
letters did not decode to an ASCII character and were discarded. By looking at
the possible decoded characters column, it is fairly easy to identify the decoded
message as “THE BEGINNING OF THAT PERFORMANCE”.
The first sentence we decoded was "THE BEGINNING OF THAT PERFORMANCE
WAS CLEAR ENOUGH." An internet search of that brought up the text of Thomas
Pynchon's "The Crying of Lot 49". We decoded other sentences to confirm that the
decoded message was the text of that book. We had finally decoded the San Jose
Semaphore!

Even though we had figured out the message, there were still a few loose ends to clean
up. First, we still didn't know which keyword letter to use for encoding each character of
the text. After some investigation we found out that the keyword letters go sequentially
through the keyword, but the direction changed often.

For instance, if the keyword is FACTION, the order of the letters used to encode each
character of the message may be:
 FACTCTCTIONFACTIOITCAFAFNOI

Further investigation showed that the direction changes whenever sound5 > sound 6. We
were now able to automatically decode the entire message, and it was indeed the entire
text of "The Crying of Lot 49". We also noticed that each data block contained one
paragraph of the book, which explained why some blocks were very short and others
were quite long.

Conclusions
Decoding the Semaphore was an interesting challenge, and fun, too. The aesthetics of the
art itself made it much more interesting than a standard puzzle, and we hope you get a
chance to see the Semaphore in action in its San Jose location or in the Simulcast at
www.sanjosesemaphore.org.
             San Jose Semaphore: The Solution
                                 Ben Rubin
                           benrubin@earstudio.com
                               August 7, 2007



She looked down a slope, needing to squint for the sunlight, onto a vast
sprawl of houses which had grown up all together, like a well-tended crop,
from the dull brown earth; and she thought of the time she'd opened a
transistor radio to replace a battery and seen her first printed circuit. The
ordered swirl of houses and streets, from this high angle, sprang at her
now with the same unexpected, astonishing clarity as the circuit card had.
Though she knew even less about radios than about Southern Californians,
there were to both outward patterns a hieroglyphic sense of concealed
meaning, of an intent to communicate. There'd seemed no limit to what
the printed circuit could have told her (if she had tried to find out); so in
her first minute of San Narciso, a revelation also trembled just past the
threshold of her understanding.

Excerpt from Thomas Pynchon’s 1966 novel, The Crying of Lot 49
Introduction

San José Semaphore, by artist Ben Rubin, is a public artwork commissioned by Adobe
Systems Incorporated in collaboration with the City of San Jose’s Office of Cultural
Affair’s Public Art Program.

Located within the top floors of Adobe’s Almaden Tower headquarters in San José,
California, San José Semaphore is a multi-sensory kinetic artwork that illuminates the
San José skyline with the transmission of a coded message. San José Semaphore’s four
ten-foot wide illuminated disks rotate every 7.2 seconds, engaging viewers with a steady,
glowing, and purposefully moving presence. The artwork’s illuminated disks perform a
kind of mechanical dance as a method of communication. A low-power radio broadcast
provides a soundtrack that is audible within 2-3 blocks of the Almaden Tower on AM
1680. An online simulcast provides a way to see and hear the piece remotely.

Cracking the coded message has been posed as a challenge for the public. This document
presents the solution to the code, revealing both the content of the Semaphore’s message,
and the means by which it was encrypted.

The Message Content Revealed

For the past year, the San Jose semaphore has been broadcasting the
full text of Thomas Pynchon’s 1966 novel, The Crying of Lot 49.
Given the artwork's location (the heart of Silicon Valley) and concept (a semaphore),
there was really only one logical choice for the text: Thomas Pynchon's The Crying of
Lot 49. Although he wrote the book in the mid 1960s, Pynchon's setting is
a fictional California city filled with high-tech industrial parks and the kind of
engineering sub-culture that we now associate with the Silicon Valley. The book follows
the heroine's discovery of latent symbols and codes embedded in this landscape and in
the local culture. Is there a message here, she wonders, and what are these symbols
trying to tell me? At its heart, San Jose Semaphore is an expression of what Pynchon
calls "an intent to communicate."

The Code

There were three main principles that went into the design of the code. The first was
“crackability.” I wanted the code to be hard to crack, but not so hard that a dedicated,
knowledgeable person (or team) could not break it within a year or two. The second
principle had to do with the visual and audible design of the artwork. I wanted the nature
of the code to relate to the form of the artwork (the four turning discs). The third
principle was to make the code human-readable. Once its structure is revealed, the cipher
can be read with the naked eye, recorded with pencil and paper, and decoded readily (if
somewhat laboriously) by hand. This human-scale communication is critical to the
artwork’s concept.
The 256-letter Alphabet

The nature of the discs themselves dictated the “alphabet” I would use for my ciphertext.
Each of the discs can assume four easily distinguishable positions: horizontal, left-
leaning diagonal, vertical, and right-leaning diagonal. With these four positions, and four
independent discs, the Semaphore as a whole has a total of 256 possible combinations of
disc positions, or values (see Fig. 1). These 256 combinations form the alphabet for the
code transmission.




   Fig. 1: The Semaphore alphabet, showing all possible disc combinations and their assigned values
First Layer: Prolog

The code is designed with two layers. The first layer is called the “prolog.” The prolog
does not contain any actual message content, but it marks the start of each paragraph of
the novel and, most important, the prolog contains the key to decoding the paragraph that
follows.

The prolog is short, and it is always the same length (41 characters). This is in contrast to
the paragraphs themselves, which vary considerably in length. The prolog always begins
with the words “START MESSAGE TRANSMISSION,” followed by a three digit
number, a colon, and a seven-letter word. The number is the paragraph number of the
text to follow, and the seven-letter word is the key for decoding that paragraph. For
example, the plaintext of the prolog for paragraph 226 looks like this:

       START MESSAGE TRANSMISSION 226:FACTION
       (“226” is the paragraph number, and “FACTION” is the key for deciphering the paragraph)


The prolog is weakly encrypted using a simple substitution cipher (see appendix). My
idea was to make the prolog relatively easy to crack (at least two teams got this far), and
thereby offer encouragement to anyone trying to break the code. The prolog also gives
away a critical piece of information: the key to cracking the text that follows. This key
is only of value, however, if the code breaker can figure out how to actually use it to
decode the paragraph; revealing the key alone does not reveal this mechanism.

It is easy to identify the prolog when watching and/or listening to the Semaphore. During
the transmission of paragraph content (which is most of the time), each of the
Semaphore’s discs spins either clockwise or counterclockwise, and there is no predictable
pattern to the direction of spin. During the prolog, all the discs spin only clockwise.

Also, during paragraph transmission, a variety of pitches are used for instrumental
sounds, tones and a signing voice (again, in an unpredictable pattern) to generate an
unfolding melody. During the prolog, there is no variation in any of the sounds’ pitches,
and the melody becomes a kind of monotone chant (more details about the sound are at
the end of this paper).

This combination of audible and visible markers was deigned to make the prolog easy to
notice, offering a first foothold for would-be code breakers.

The Keys

The keys are seven letter words (the same number of letters as in “PYNCHON”) taken in
order of appearance from James Joyce’s Ulysses. I have used text from this novel in
previous artworks, and the unique (and sometimes cryptic) language seemed a good fit
for this cryptographic purpose. The book is also long and filled with exotic vocabulary,
and so it contains a bounty of unique seven-letter words (more than enough to provide a
unique key for each paragraph). Potential keys containing the letters H, Q, or X were not
used for reasons that are explained below.
Here are examples of seven-letter keys from Ulysses:

SINGLED                   ABDOMEN                   WRONGED                    BRUCKEN                   ORDINAL
FIGURED                   MASTINO                   BYGONES                    INSULAR                   ROMANUS
BRACING                   UNSCREW                   BASKING                    ALIMENT                   CRAWLED
ROAMING                   LEVYING                   DICTUMS                    SOLVENT                   COGNATE
CARKING                   IRATELY                   RACKING                    DENSITY                   FRONTAL
MARKING                   INDULGE                   LAUREIS                    FAILURE                   SECULAR
GUNBOAT                   UMBRAGE                   TOMKINS                    FIRSTLY                   AGONIES
LASTING                   AUDIBLY                   NOVELTY                    TANDEMS                   LOCATED
DURANCE                   AROUSED                   FOISTED                    NEBUKIM                   BREVITY
OUTLINE                   CUDGELS                   OBVIATE                    MENDOZA                   ORGANIC

I had originally planned to use key words derived directly from The Crying of Lot 49
itself. I changed my plans, however, after I tried entering an arbitrary selection of four
such words into Google, and links to The Crying of Lot 49 came up immediately. I then
realized that I would need to find keys from a different source. I knew Ulysses would
similarly reveal itself to the code-breakers, but it seemed a worthy red-herring.

Second Layer: The Text

The second layer of the code (in which the 836 paragraphs of The Crying of Lot 49 are
actually encoded), is a variant on the classic Vigenère cipher, which was thought to be
unbreakable for nearly 300 years after its publication in 1585. As described on
Wikipedia:

          In the Vigenère cipher, the first row of the tableau is filled out with a copy of the
          plaintext alphabet, and successive rows are simply shifted one place to the left. (Such a
          simple tableau is called a tabula recta, and mathematically corresponds to adding the
          plaintext and key letters, modulo 26.) A keyword is then used to choose which ciphertext
          alphabet to use. Each letter of the keyword is used in turn, and then they are repeated
          again from the beginning. So if the keyword is 'CAT', the first letter of plaintext is
          enciphered under alphabet 'C', the second under 'A', the third under 'T', the fourth under
          'C' again, and so on. In practice, Vigenère keys were often phrases several words long.

The Semaphore code varies from the classic Vigenère in a two ways: first, the successive
rows in the table are not merely shifted, but instead are the product of a simple
calculation (described below), and second, where a classic Vigenère cipher always rotates
the key forward, the Semaphore code may shift the key in either direction at each step.
They key to the shift direction is hidden in the visible rotation of the discs (and in the
corresponding sound).

The direction for the next rotation of the key is encoded in the rotation of the
Semaphore’s third disc (the third from the left). Whenever this disc rotates clockwise,
the key must be rotated forward, and when the disc rotates counterclockwise, the key
must be rotated backwards.1


1
 In order to make frequency analysis of the ciphertext a bit more difficult, the rotation directions were generated using
an algorithm that maximized the diversity of plaintext / key letter combinations.
The audio contains a parallel indicator: there are two short tones in the sequence that are
in quick succession. When the third disc rotates clockwise, these tones always rise (a low
tone, followed quickly by a higher one); when the third disc rotates counterclockwise, the
tones fall. The tones thus redundantly encode the instruction for the key rotation.

The Key Table

The Semaphore’s version of the Vigenère “tableau” is a key table (see Fig. 2) that assigns
64 possible plain text characters (26 letters, 10 numbers, plus most common punctuation,
all in standard ASCII order) to the 256 possible Semaphore values based on 23 key
values A through Z (H, P, and X are omitted). The ciphertext (output) values are
generated by multiplying the input value by the key value, then taking MOD 256 of the
result. Note that H, P, and X are not valid key values, because they end up with duplicate
ciphertext entries.
          1     2     3     4     5     6     7     8     9    10    11    12    13    14    15    16    17    18    19    20    21    22    23    24    25    26
          A     B     C     D     E     F     G     H     I     J     K     L     M     N     O     P     Q     R     S     T     U     V    W      X     Y     Z
 1        1     2     3     4     5     6     7     8     9    10    11    12    13    14    15    16    17    18    19    20    21    22    23    24    25    26
 2   !    2     4     6     8    10    12    14    16    18    20    22    24    26    28    30    32    34    36    38    40    42    44    46    48    50    52
 3   "    3     6     9    12    15    18    21    24    27    30    33    36    39    42    45    48    51    54    57    60    63    66    69    72    75    78
 4   #    4     8    12    16    20    24    28    32    36    40    44    48    52    56    60    64    68    72    76    80    84    88    92    96   100   104
 5   $    5    10    15    20    25    30    35    40    45    50    55    60    65    70    75    80    85    90    95   100   105   110   115   120   125   130
 6   %    6    12    18    24    30    36    42    48    54    60    66    72    78    84    90    96   102   108   114   120   126   132   138   144   150   156
 7   &    7    14    21    28    35    42    49    56    63    70    77    84    91    98   105   112   119   126   133   140   147   154   161   168   175   182
 8        8    16    24    32    40    48    56    64    72    80    88    96   104   112   120   128   136   144   152   160   168   176   184   192   200   208
 9   (    9    18    27    36    45    54    63    72    81    90    99   108   117   126   135   144   153   162   171   180   189   198   207   216   225   234
10   )   10    20    30    40    50    60    70    80    90   100   110   120   130   140   150   160   170   180   190   200   210   220   230   240   250     4
11   *   11    22    33    44    55    66    77    88    99   110   121   132   143   154   165   176   187   198   209   220   231   242   253     8    19    30
12   +   12    24    36    48    60    72    84    96   108   120   132   144   156   168   180   192   204   216   228   240   252     8    20    32    44    56
13   ,   13    26    39    52    65    78    91   104   117   130   143   156   169   182   195   208   221   234   247     4    17    30    43    56    69    82
14   -   14    28    42    56    70    84    98   112   126   140   154   168   182   196   210   224   238   252    10    24    38    52    66    80    94   108
15   .   15    30    45    60    75    90   105   120   135   150   165   180   195   210   225   240   255    14    29    44    59    74    89   104   119   134
16   /   16    32    48    64    80    96   112   128   144   160   176   192   208   224   240     0    16    32    48    64    80    96   112   128   144   160
17   0   17    34    51    68    85   102   119   136   153   170   187   204   221   238   255    16    33    50    67    84   101   118   135   152   169   186
18   1   18    36    54    72    90   108   126   144   162   180   198   216   234   252    14    32    50    68    86   104   122   140   158   176   194   212
19   2   19    38    57    76    95   114   133   152   171   190   209   228   247    10    29    48    67    86   105   124   143   162   181   200   219   238
20   3   20    40    60    80   100   120   140   160   180   200   220   240     4    24    44    64    84   104   124   144   164   184   204   224   244     8
21   4   21    42    63    84   105   126   147   168   189   210   231   252    17    38    59    80   101   122   143   164   185   206   227   248    13    34
22   5   22    44    66    88   110   132   154   176   198   220   242     8    30    52    74    96   118   140   162   184   206   228   250    16    38    60
23   6   23    46    69    92   115   138   161   184   207   230   253    20    43    66    89   112   135   158   181   204   227   250    17    40    63    86
24   7   24    48    72    96   120   144   168   192   216   240     8    32    56    80   104   128   152   176   200   224   248    16    40    64    88   112
25   8   25    50    75   100   125   150   175   200   225   250    19    44    69    94   119   144   169   194   219   244    13    38    63    88   113   138
26   9   26    52    78   104   130   156   182   208   234     4    30    56    82   108   134   160   186   212   238     8    34    60    86   112   138   164
27   :   27    54    81   108   135   162   189   216   243    14    41    68    95   122   149   176   203   230     1    28    55    82   109   136   163   190
28   ;   28    56    84   112   140   168   196   224   252    24    52    80   108   136   164   192   220   248    20    48    76   104   132   160   188   216
29   <   29    58    87   116   145   174   203   232     5    34    63    92   121   150   179   208   237    10    39    68    97   126   155   184   213   242
30   =   30    60    90   120   150   180   210   240    14    44    74   104   134   164   194   224   254    28    58    88   118   148   178   208   238    12
31   >   31    62    93   124   155   186   217   248    23    54    85   116   147   178   209   240    15    46    77   108   139   170   201   232     7    38
32   ?   32    64    96   128   160   192   224     0    32    64    96   128   160   192   224     0    32    64    96   128   160   192   224     0    32    64
33   @   33    66    99   132   165   198   231     8    41    74   107   140   173   206   239    16    49    82   115   148   181   214   247    24    57    90
34   A   34    68   102   136   170   204   238    16    50    84   118   152   186   220   254    32    66   100   134   168   202   236    14    48    82   116
35   B   35    70   105   140   175   210   245    24    59    94   129   164   199   234    13    48    83   118   153   188   223     2    37    72   107   142
36   C   36    72   108   144   180   216   252    32    68   104   140   176   212   248    28    64   100   136   172   208   244    24    60    96   132   168
37   D   37    74   111   148   185   222     3    40    77   114   151   188   225     6    43    80   117   154   191   228     9    46    83   120   157   194
38   E   38    76   114   152   190   228    10    48    86   124   162   200   238    20    58    96   134   172   210   248    30    68   106   144   182   220
39   F   39    78   117   156   195   234    17    56    95   134   173   212   251    34    73   112   151   190   229    12    51    90   129   168   207   246
40   G   40    80   120   160   200   240    24    64   104   144   184   224     8    48    88   128   168   208   248    32    72   112   152   192   232    16
41   H   41    82   123   164   205   246    31    72   113   154   195   236    21    62   103   144   185   226    11    52    93   134   175   216     1    42
42   I   42    84   126   168   210   252    38    80   122   164   206   248    34    76   118   160   202   244    30    72   114   156   198   240    26    68
43   J   43    86   129   172   215     2    45    88   131   174   217     4    47    90   133   176   219     6    49    92   135   178   221     8    51    94
44   K   44    88   132   176   220     8    52    96   140   184   228    16    60   104   148   192   236    24    68   112   156   200   244    32    76   120
45   L   45    90   135   180   225    14    59   104   149   194   239    28    73   118   163   208   253    42    87   132   177   222    11    56   101   146
46   M   46    92   138   184   230    20    66   112   158   204   250    40    86   132   178   224    14    60   106   152   198   244    34    80   126   172
47   N   47    94   141   188   235    26    73   120   167   214     5    52    99   146   193   240    31    78   125   172   219    10    57   104   151   198
48   O   48    96   144   192   240    32    80   128   176   224    16    64   112   160   208     0    48    96   144   192   240    32    80   128   176   224
49   P   49    98   147   196   245    38    87   136   185   234    27    76   125   174   223    16    65   114   163   212     5    54   103   152   201   250
50   Q   50   100   150   200   250    44    94   144   194   244    38    88   138   188   238    32    82   132   182   232    26    76   126   176   226    20
51   R   51   102   153   204   255    50   101   152   203   254    49   100   151   202   253    48    99   150   201   252    47    98   149   200   251    46
52   S   52   104   156   208     4    56   108   160   212     8    60   112   164   216    12    64   116   168   220    16    68   120   172   224    20    72
53   T   53   106   159   212     9    62   115   168   221    18    71   124   177   230    27    80   133   186   239    36    89   142   195   248    45    98
54   U   54   108   162   216    14    68   122   176   230    28    82   136   190   244    42    96   150   204     2    56   110   164   218    16    70   124
55   V   55   110   165   220    19    74   129   184   239    38    93   148   203     2    57   112   167   222    21    76   131   186   241    40    95   150
56   W   56   112   168   224    24    80   136   192   248    48   104   160   216    16    72   128   184   240    40    96   152   208     8    64   120   176
57   X   57   114   171   228    29    86   143   200     1    58   115   172   229    30    87   144   201     2    59   116   173   230    31    88   145   202
58   Y   58   116   174   232    34    92   150   208    10    68   126   184   242    44   102   160   218    20    78   136   194   252    54   112   170   228
59   Z   59   118   177   236    39    98   157   216    19    78   137   196   255    58   117   176   235    38    97   156   215    18    77   136   195   254
60   [   60   120   180   240    44   104   164   224    28    88   148   208    12    72   132   192   252    56   116   176   236    40   100   160   220    24
61   \   61   122   183   244    49   110   171   232    37    98   159   220    25    86   147   208    13    74   135   196     1    62   123   184   245    50
62   ]   62   124   186   248    54   116   178   240    46   108   170   232    38   100   162   224    30    92   154   216    22    84   146   208    14    76
63   ^   63   126   189   252    59   122   185   248    55   118   181   244    51   114   177   240    47   110   173   236    43   106   169   232    39   102
64   _   64   128   192     0    64   128   192     0    64   128   192     0    64   128   192     0    64   128   192     0    64   128   192     0    64   128



Fig. 2: The Semaphore’s Vigenère “square” (key table). Plain text (input) values are shown at left in
yellow, and key values are shown across the top, also in yellow.
I chose this scheme as a way to expand my 64-character plaintext alphabet (26 letters, 10
numbers, and assorted punctuation) to use the entire 256-member Semaphore alphabet,
ensuring that there would be as much visual variety as possible in the movement of the
discs.

The Soundtrack

In addition to its aesthetic value, the soundtrack adds a redundant means of receiving the
code transmission. The Semaphore is designed so that the code can be fully transmitted
(and cracked) based solely on the visible movements, or solely on the sound — all the
critical information is fully represented in each modality.

The voices communicate the positions of the discs when they come to rest. The first
voice, a distant spoken voice, speaks a letter of the NATO phonetic alphabet (Alfa,
Bravo, Charlie up through the alphabet to Papa), and the second voice sings a number
between one and sixteen. The first voice indicates the positions of the leftmost pair of
discs, and the second voice indicates the positions for the rightmost pair. Each letter or
number corresponds to a unique combination of positions as shown in the table below
(Fig. 3).




                              Fig. 3: The key to decoding the voices

As discussed above, two of the tonal sounds in the soundtrack provide a redundant
indication of the spin direction for the third disc (and hence, provide parallel instructions
for rotating the substitution key). The other sounds, like the spin directions of the other
three discs, are purely ornamental.

The aesthetic of the soundtrack is based on my childhood memories of listening to the
shortwave radio; for me, they evoke the mysterious cold-war “numbers stations”
broadcasts, as well as the regular pulsing rhythm of National Bureau of Standards’
Atomic Clock.
APPENDIX: The simple substitution cipher used for the prolog


SEMAPHORE         PLAINTEXT                  SEMAPHORE         PLAINTEXT
  ----                                         |---                @
  /-\-                 !                       \-\-                A
  |-|-                 "                       --|-                B
  \-/-                 #                       /-/-                C
  -\--                 $                       |\--                D
  /\\-                 %                       \\\-                E
  |\|-                 &                       -\|-                F
  \\/-                                         /\/-                G
  -|--                 (                       ||--                H
  /|\-                 )                       \|\-                I
  |||-                 *                       -||-                J
  \|/-                 +                       /|/-                K
  -/--                 ,                       |/--                L
  //\-                 -                       \/\-                M
  |/|-                 .                       -/|-                N
  \//-                 /                       ///-                O
  \---                 0                       /---                P
  --\-                 1                       |-\-                Q
  /-|-                 2                       \-|-                R
  |-/-                 3                       --/-                S
  \\--                 4                       /\--                T
  -\\-                 5                       |\\-                U
  /\|-                 6                       \\|-                V
  |\/-                 7                       -\/-                W
  \|--                 8                       /|--                X
  -|\-                 9                       ||\-                Y
  /||-                 :                       \||-                Z
  ||/-                 ;                       -|/-                [
  \/--                 <                       //--                \
  -/\-                 =                       |/\-                ]
  //|-                 >                       \/|-                ^
  |//-                 ?                       -//-                _

				
DOCUMENT INFO
Categories:
Tags:
Stats:
views:22
posted:11/19/2012
language:
pages:26