In this paper we expand on prior research into the use of Continuous Time Recurrent Neural Networks (CTRNNs) as evolvable generators of musical structures such as audio waveforms. This type of neural network has a compact structure and is capable of producing a large range of temporal dynamics. Due to these properties, we believe that CTRNNs combined with evolutionary algorithms (EA) could offer musicians many creative possibilities for the exploration of sound. In prior work, we have explored the use of interactive and target-based EA designs to tap into the creative possibilities of CTRNNs. Our results have shown promise for the use of CTRNNs in the audio domain. However, we feel neither EA designs allow both open-ended discovery and effective navigation of the CTRNN audio search space by musicians. Within this paper, we explore the possibility of using novelty search as an alternative algorithm that facilitates both open-ended and rapid discovery of the CTRNN creative search space.