Original paper
Trijrtrm.nr.ijt
I MTDF.M
A Innrnal of
Journal of Microelectronics, Electronic Components and Materials Vol. 42, No. 2 (2012), 115 - 122
Time-transfer and synchronization equipment for high-performance parti^cle accelerators
Jurij Tratnik,1,2 Primož Lemut2 and Matjaž Vidmar1,2
1 University of Ljubljana, Faculty of Electrical Engineering, Ljubljana, Slovenia
2COBIK - Centre of excellence for Biosensors, Instrumentation and Process Control, Solkan,
Slovenia
Abstract: This article talks about accurate time-transfer and synchronization in particle accelerator facilities. Simple industrial solutions like TTL-level pulses and 10 MHz reference clocks distributed over conventional coaxial cables are both much worse than the theoretical limits for timing accuracy, neither do satisfy the requirements of high-performance particle accelerators. Several different approaches to the time-transfer and synchronization problem have been implemented elsewhere. Yet their accuracy is still far from theory limits in most cases. Direct comparisons are difficult due to the very different formats in which the results were published. This article attempts to describe the theoretical backgrounds of timing first, considering natural physical limits rather than particular implementations. Next the limitations of different implementation technologies are examined including practical but very important issues like technology maturity, component availability and obsolescence. Optical-fiber technology is then examined into detail, including all limitations and interface problems with other equipment at both ends of the synchronization links. Different practical approaches using optical fiber as the transmission medium for timing and synchronization are also discussed. At the end, our implementation of a prototype compensated CW modulation timing system is described into detail including the stability-measurement results obtained in a real accelerator tunnel. Industrialization and further developments of our system are also presented. The latter include single-fiber bidirectional operation, already showing further improved timing stability in non-perfect laboratory conditions and a possible development of a high-stability electro-optical master clock oscillator. Our conclusion is that besides the theoretical background a detailed knowledge of many different technologies is required to combine them in a topperformance timing system.
Key words: Timing system, synchronization, technology, optical fiber
Prenos takta in sinhronizacijska oprema za visoko-zmogljive pospeševalnike osnovnih delcev
Povzetek: Predstavljeni članek govori o točnem prenosu časa in sinhronizaciji pospeševalnikov osnovnih delcev. Enostavne industrijske rešitve kot na primer prenos TTL impulzov ali 10 MHz referenčnega takta po konvencionalnih koaksialnih kablih še zdaleč ne dosegajo teoretičnih omejitev točnosti niti ne zadoščajo zahtevam visoko-zmogljivih pospeševalnikov. Za rešitev tega problema je poznanih in implementiranih že kar nekaj prenosnih sistemov. Njihova točnost je v večini primerov žal še vedno daleč od teoretičnih mej. Zahtevne so tudi neposredne primerjave obstoječih sistemov, saj so merilni rezultati predstavljeni v različnih formatih in izmerjeni na različne načine. V članku je sprva opisano teoretično ozadje prenosa točnega časa. Pri tem je bolj kot na dejanskih prenosnih sistemih poudarek na naravnih fizikalnih zakonih in omejitvah. V nadaljevanju sledi pregled različnih tehnologij prenosa, njihovih praktičnih omejitev, zrelosti tehnologije ter dobavljivosti komponent. Uporaba optičnih, vlakenskih tehnologij je nato predstavljena bolj podrobno. Opisane so omejitve ter vmesniški problemi z ostalo opremo na obeh koncih sinhronizacijske povezave. Predstavljeni so tudi različni praktični načini sinhronizacije z uporabo optičnih vlaken kot prenosnim medijem. Na koncu je predstavljen tudi izdelani prototip za stabiliziran prenos moduliranega optičnega signala preko optičnih vlaken. Podane so meritve stabilnosti, izmerjene v pospeševalniškem tunelu ter nakazane bodoče industrijske izboljšave. Ena od njih vključuje prenos mikrovalovnega takta po enem samem stabiliziranem optičnem vlaknu. Prvi testi že kažejo izboljšanje stabilnosti pri prenosu, omenjeni princip pa je mogoče uporabiti tudi pri razvoju visoko-stabilnega elektro-optičnega oscilatorja. Iz navedenega je mogoče zaključiti, da je poleg teoretičnega ozadja, potrebno zelo podrobno poznavanje različnih vrst tehnologij, ki na koncu sestavljajo dovršen sinhronizacijski sistem.
Ključne besede: Sistem za prenos časa, sinhronizacija, tehnologija, optično vlakno
' Corresponding Author's e-maii:jurij.tratnik@fe.uni-ij.si
1 Introduction -jitter and drift
ung accuracy:
One of the design requirements of modern particle accelerators is precise timing and synchronization of the machine components and user experiments at different physical locations [1, 2]. Before designing a high-accuracy timing system, some basic definitions and limitations need to be considered first to select the most suitable technology for a certain task. Timing accuracy includes at least two different specifications, short-term inaccuracy described as phase noise or jitter and long-term inaccuracy described as wander or drift.
1.1 Jitter
Jitter is mainly caused by random noise added to the timing (clock) signal as shown in Fig. 1.
Figure 1: Signal-to-noise ratio and timing jitter.
The resulting jitter At is a function of the clock period T (or frequency f) and the signal-to-noise ratio.
(1)
Please note that the signal-to-noise ratio may be expressed in many different physical quantities. Voltages are usually used for electrical signals. Signal and noise powers have a more general meaning. Noise may come from many different sources. Natural noise sources like thermal noise or quantum noise shown in Fig. 2 can not be avoided.
Noise power is usually reduced by narrow-band filtering with resonators having a quality Q. The filter bandwidth B=f/Q is orders of magnitude smaller than the clock frequency f. In narrow-band systems the noise power is simply described by its spectral density Ng. In an electrical timing system operating in the radio/microwave frequency range, natural noise is manly thermal noise. The resulting jitter is inversely proportional to the square root of the frequency.
Figure 2: Thermal and quantum noise.
(2)
Therefore the performance of such a timing system improves at higher frequencies provided that all of the remaining design parameters remain the same. In an optical timing system operating at visible light or near IR, natural noise is mainly quantum (shot) noise. The resulting jitter is frequency-independent.
(3)
Please note that the resonator Q is limited by its mechanical stability and thermal drift in all cases. Therefore there is no particular theoretical advantage in using optical or even higher frequencies for timing.
1.2 Drift
Drift is mainly caused by temperature and other environmental variations, by power-supply variations, by component imperfections like AM-to-PM conversion, by component degradation & aging etc. Drift is not directly related to the theoretical design parameters of a timing system like its clock frequency. On the other hand, drift is related to the technologies used and in particular to their state of maturity.
2 Component Technologies
20 or 30 years ago it was considered that component reliability could be designed into the component itself by very simple means. For example by choosing the right manufacturing process or better packaging for indus-trialgrade parts. Additional inspections and screening was used for military/aerospace-grade parts. Econo-
mies of scale first hit the personal-computer market. In just 10 years, the personal computer evolved in a top-technology item in 2000. After 2000 inexpensive personal computers with GPU computing (GPU - Graphic Processing Units) quickly displaced much more expensive workstations and other computing hardware [3].
A quality quantum leap was achieved in mobile telephony. Highly-reliable mobile phones had to be manufactured in large volumes at consumer prices but respecting industrial or military quality standards. The distinction among consumer, industrial and military grades suddenly disappeared.
The reliability of modern electronic and optical components is no simple matter. The most obvious example are the failure mechanisms of semiconductor lasers. Individual component screening like performed on military-grade parts many years ago does not help. The correct answer is learning from mass production in million series and correcting the manufacturing process all of the time [4]. This simply means that specialized components can no longer be produced, at least not at the reliability levels that are obvious today.
An increasingly important component issue is complexity. Component complexity is not limited to software, although the latter is the most obvious complexity issue. The hardware design of most electronic and optical components has become so complex that the design of specialized components can no longer be afforded by a small group of engineers, especially not in the limited amount of time allowed by practical projects.
Finally, a completely new problem appeared in the 21"* century called component obsolescence. Economies of scale also dictate that the production of a particular component is dropped as soon as the latter becomes obsolete. Due to the component complexity it is usually difficult to have it manufactured at a different location or find similar plug-in replacements.
As shown in Table 1, a design engineer has to be extremely careful while choosing the technologies for a new product. While there are some mass-produced and inexpensive components with fantastic performance, there are many technology failures as well. The support to some frequency ranges and applications may be missing. Even successful technologies may be affected by component obsolescence.
Table 1: Technology successes and failures.
Technology successes	Technology failures
Analog radio/microwave electronics	Millimeter-wave electronics
High-speed digital electronics	RF micro-electromechanical devices (RF MEMS)
(Electronic) digital signal processing (DSP)	Long-wave (thermal) IR optics
Silica-glass optical fiber (waveguide)	Fiber-optic laser sources (oscillators): CW, pulsed, mode-locked
Semiconductor lasers, modulators and photo-detectors	Optical signal processing (holography, nonlinear optics)
Erbium-doped fiber laser amplifier (EDFA)	Optical computing
3 Fiber-optic technology
The availability of inexpensive optical fiber triggered the development of all related components: splices, connectors, LED and LASER transmitters, modulators, isolators, circulators, PIN and avalanche photo-diode receivers and even Erbium-doped fiber-optic LASER amplifiers [5]. All these new components changed the meaning of the word optics. Traditionally, optics meant bulk optics with lenses, mirrors and various cumbersome components on optical benches in clean-room environments, requiring precise handling and extremely sensitive to dirt, moisture and vibration. On the other hand, optical fiber and all fiber-optic components are designed right from the beginning for simple handling, stable and reliable operation in the most unforgiving environments ranging from the ocean floor [6] up to the geostationary orbit [7].
Yet optical fiber does have some limitations that have to be understood in order to design a successful system. The most obvious limitation is the frequency or wavelength range. Silica glass only works in the visible and near-infrared range. Other optical-fiber limitations are shown in Fig. 3.
Standard Telecom G.652 single-mode optical fibers have chromatic dispersion due to both waveguide and material effects [8]. Rotationally-symmetrical fibers do not maintain polarization. The random coupling between the two degenerate fundamental modes causes Polarization mode dispersion (PMD). The refractive index of silica glass has a large temperature coefficient. The latter may become unpredictable and up to an order of magnitude larger due to improper (tight) cabling
Figure 3: Single-mode optical-fiber properties.
like used in patchcords or FTTH cables [9, 10]. Although much less sensitive to vibration than bulk optics, microphonics may still represent a problem in high-accuracy fiber-optic systems.
A major disadvantage of fiber-optic technology is fiber non-linearity. In most cases the fiber signal power is limited to about 100 mW due to both Kerr and Raman effects. Coherent systems using very narrow-band optical signals may be limited to power levels below 1 mW due to Brillouin scattering.
4 Optical timing systems
The typical distance to be covered by a timing system in a particle accelerator ranges from a few hundred meters to a few ten kilometers. At these distances coaxial cable becomes bulky and lossy as the clock frequency is increased to improve accuracy [11]. Optical fiber is an excellent substitute offering low loss at light-wave frequencies. Usually, no amplification is required in an optical-fiber system up to at least 50 km. As shown in Fig. 4, optical timing systems include:
•	Optical CW systems,
•	Pulsed systems and
•	CW modulation systems.
4.1 Optical CW systems
Optical CW systems should offer the highest resolution and accuracy due to the high clock-signal frequency. Unfortunately the resulting 5.16 fs timing ambiguity at 194 THz is too small for practical applications. Vibrations might cause cycle slips and the system phase can never be recovered after any power down. The extremely narrowband optical signal causes interferometric noise and triggers Brillouin scattering. Polarization changes and PMD in optical fibers are a big problem. Last but not least, no user equipment is currently available that
Figure 4: Optical timing systems.
could use the timing information of an optical carrier directly. However, an optical CW signal can be used for optical-link stabilization with an interferometer setup, as shown in [12, 13].
4.2	Pulsed systems
Pulsed systems use an optical carrier while the timing information is carried in the pulse envelope [14, 15]. The spectrum of the latter may extend in the millimeter/ long-wave IR to offer precise timing and avoid phase ambiguity at the same time. Unfortunately, pulsed systems are affected by fiber non-linearity, chromatic dispersion and PMD. Compensation of the fiber thermal coefficient is very difficult. Pulsed systems are well understood by the user community although optical signal processing is not here yet and the electrical signal-to-noise ratio from a photodetector may be poor.
4.3	CW modulation systems
Replacing a few large pulses with many more smaller pulses results in a CW modulation system [16-19]. Fiber nonlinearity can be avoided due to the lower peak power. The signal distortion caused by chromatic dispersion and PMD is much less critical than in pulsed systems. Standard, mass-produced, inexpensive, highperformance and high-reliability Telecom components may be used. The electrical input and output of CW modulation systems interface to user equipment directly just like coaxial cable. Unfortunately, the timing jitter of CW modulation systems is rather poor. Most of the signal-to-noise degradation comes from the impedance mismatch between the photo-diode and following electrical amplifier as shown in Fig. 5.
Assuming an electrical noise temperature of 7=300 K and a parasitic capacitance of C=1 pF (sum of photodiode and amplifier input) results in a noise voltage of
Figure 5: Photo-diode receiver.
UNeff=25.7 ^Veff (from Eq.(4)) independent of the frequency range.

■ ^ - 25.7m F.
eir
(4)
The timing jitter can be reduced by averaging many small pulses. In other words, each small pulse just adds a small amount of momentum to a large flywheel. In electrical terms the flywheel is a narrow band-pass filter. The electrical flywheel is built as a high-Q resonator or VCXO PLL as shown in Fig. 6.
Figure 6: CW modulation system with flywheel.
A high-Q resonator allows a bandwidth reduction in the range between 104 and 106. The jitter reduction goes with the square root of the bandwidth reduction, resulting in a jitter reduction factor between 100 and 1000.
In a CW modulation system, the optical carrier is not coherent with the timing modulation. Therefore the optical carrier frequency may be modulated to avoid Brillouin scattering. Even more important, changing the optical carrier frequency together with the fiber chromatic dispersion can be used for small adjustments of the group velocity and overall system delay.
As shown in Fig. 7, delay-variation compensation can be achieved in different ways. Fast variations like vibrations can be compensated by electrically tuning the DFB LASER over a restricted bandwidth of just +/-0.1
nm in a very fast way (t =1 ^s). Medium-speed variations can be corrected by thermally tuning the DFB LASER over +/-2 nm (t=1s). Slow variations like temperature changes of the transmission fiber can be adjusted by heating or cooling a spool of compensating fiber (t =100 s).
Figure 7: Delay-variation compensation techniques.
5 Prototype compensated CW modulation system
A prototype CW modulation system Libera Sync was built by Instrumentation Technologies [20] for the accelerator "FERMI" distributing a 3 GHz clock over distances up to 300 m. A fiber pair made of two identical optical fibers in the same cable was used for clock distribution. In this way the actual delay could be measured at any time and any variations compensated immediately.
Figure 8: Compensated CW modulation system.
The block diagram of the 3 GHz compensated CW modulation system is shown in Fig. 8. A flywheel resonator with the Q=104 was used to reduce the timing
jitter. For the phase noise evaluation of the system, a signal source analyser (SSA) Agilent E5052B was used. The phase noise of the 3 GHz signal source is plotted in Fig. 9 (dark curve) in the frequency range from 100 Hz to 10 MHz. The phase noise after Libera Sync is plotted with a bright curve in the same graph. The noise floor of the SSA is at least 10 dB lower then measured curves. The integrated jitter increases from 12.4 fsRMS at the system input (source+SSA) to 13.4 fsRMS at the system output (source+Libera Sync+SSA). The added jitter can be then calculated as
ßttaää
-ML
^fiRMS
(5)
Figure 9: Measured phase noise and calculated jitter at 3 GHz: dark curve - the phase noise of the signal source and bright curve - the phase noise at the output of the Libera Sync timing system.
In order to compensate for long-term variations and/ or drift, Libera Sync includes three identical receiving blocks. The first receiving block is located in the transmitter to compensate for any phase-shift changes in the transmitter. The second receiving block is also located in the transmitter to compensate for transmission-line-delay variations. Finally, the third receiving block is located in the receiver and compensates for any delay variations of the flywheel resonator.
All three receiving blocks are built using matched components kept in precisely-controlled environments. Further there is a spool of two identical compensating fibers for the forward and backward signal paths that is omitted on Fig. 8 for simplicity. The spool is heated or cooled by a Peltier heat pump.
The long-term phase stability of the proposed system was measured with the the measurement setup shown in Fig. 10a. The RF master-source signal was compared to the signal transfered over the compensated optical link with an independent phase detector. The phase detector (Analog Devices AD8302) was installed in
its own, thermally-stabilized enclosure. The detected phase difference on the phase detector was measured with a Datron 1281 multimeter and sampled with a computer acquisitioning system (sampling integration time 5 seconds). The phase stability of the measurement system was 2.5 fsRMS (10 fspp), observed in 72 hours. The measured long-term drift of the timing system is shown in Fig. 10b. The measured drift is 9.5 fsRMS in 24 hours and 13.4 f^,, in 38 hours for an installed
RMS
fiber length of 360 m in the FERMI accelerator tunnel.
Figure 10: a) Long-term stability measurement setup. b) Measured long-term drift at 3 GHz of a 2-fibre system in the accelerator tunnel.
The achieved long-term phase stability and the value of added jitter are in the same range (below 10 fsRMS) as reported in other advanced optical timing systems working at the same or similar clock frequencies.
6 Further fiberoptic developments
The initial Libera Sync prototype was designed for short distances up to about 300m, where fiber-to-fiber matching is excellent and PMD is not an issue. At longer distances (3 km to 10 km) a Faraday mirror becomes necessary to use a single fiber and compensate PMD precisely as shown in Fig. 11. Some preliminary tests on a single-fiber system have been made under quite noisy laboratory conditions with several degrees centigrade of day/night temperature fluctuations. A 300 m long single-mode fiber on a spool of 1.5 m diameter was used. The measured long-term drift is 20.4 fsRMS in
13 hours as shown in Fig. 12. The signal was picked-up directly after pre-amplifier in the receiver and compared to the reference as shown in Fig. 10a. No additional signal filtering was used in this particular measurement.
Acknowlwdgements
Figure 11: PMD compensation with a Faraday mirror.
Figure 12: Measured long-term drift at 3 GHz of a single-fibre system placed in noisy laboratory conditions.
Finally, the same design and control technologies can be used to design a top performance, extremely low phase noise master oscillator as shown in Fig. 13. The combined effect of the fiber delay and flywheel resonator is an effective Q in the range between 105 and 107.
This work was supported by COBIK - Centre of excellence for Biosensors, Instrumentation and Process Control, Slovenia, Instrumentation Technologies d.d., Slovenia and Sincrotrone Trieste S.C.p.A., Italy. The authors are grateful to M. Ferianis (FERMI@Elettra) for help with long-term stability measurements.
References
Figure 13: Electro-optical master oscillator.
1.	T. Korhonen, 'Review of Accelerator Timing Systems, Proceedings of ICALEPCS99, Trieste, Italy, p. 167, 1999.
2.	M. Ferianis, 'State of the Art in High-Stability Timing, Phase Reference Distribution and Synchronization Systems, Proceedings of PAC09, Vancouver, Canada, WE3GRI02, pp. 1-5, 2009.
3.	J. D. Owens et al.,'GPU Computing, Proceedings of the IEEE, Vol. 96, No. 5, pp. 879 - 899, 2008.
4.	V. Arbet-Engels, 'Optoelectronic Analogue Signal Transfer for LHC Detectors, LEB Status Report/ RD23, CERN, Geneva, Switzerland, 1997.
5.	M. Vidmar, 'Optical-Fiber Communications: Components and Systems', Informacije MIDEM, Vol. 31, No. 4, pp. 246-251, 2001.
6.	J. Hayes, 'A History of Transatlantic Cables', IEEE Communications Magazine, Vol. 46, No. 9, pp.4248, 2008.
7.	M. N. Ott, P. Friedberg, 'Technology Validation of Optical Fiber Cables for Space Flight Environments, Proceedings of SPIE, Vol. 4216, pp. 206217, 2001.
8.	E. Udd et al., 'Fiber Optic Sensors, John Wiley and Sons, Hoboken, 2006.
9.	L. Hoffmann et al., 'Applications of Fibre Optic Temperature Measurement, Proc. Estonian Acad. Sci. Eng., Vol. 13, No. 4, pp. 363-378, 2007.
10.	M. Bousonville, J. Rausch, 'Velocity of Signal Delay Changes in Fibre Optic Cables, Proceedings of DIPAC09, Basel, Switzerland, TUPB35, pp. 248-250, 2009.
11.	S. Hunziker, V. Schlott, 'Towards an Ultra-Stable Reference Distribution for the New PSI 250 MeV Injector, Proceedings of DIPAC09, Basel, Switzerland, TUPB43, 2009.
12.	J. M. Byrd et al., in Proceedings of LINAC2006, Knoxville, Tennessee, USA, THP007, pp. 577-579, 2006.
13.	R. Wilcox et al., 'Stable Transmission of Radio Frequency Signals on Fiber Links Using Interferomet-ric Delay Sensing', Opt. Lett. Vol. 34, No. 20, pp. 3050-3052, 2009.
14.	D. D. Hudson, S. M. Foreman, S. T. Cundiff, J. Ye, 'Synchronization of Mode-Locked Femtosecond Lasers through a Fiber Link, Opt. Lett. Vol. 31, No. 13, pp. 1951-1953, 2006.
15.	J. Kim et al., 'Long-term Femtosecond Timing Link Stabilization Using a Single-Crystal Balanced Cross Correlator, Opt. Lett. Vol. 32, No. 9, pp. 10441046, 2007.
16.	J. Frisch, D. Bernstein, D. Brown, E. Cisneros, 'A High Stability, Low Noise RF Distribution System, Proceedings of PAC01, Chicago, Illinois, pp. 816818, 2001.
17.	R. S. Romaniuk et al., 'Optical Network and FPGA/ DSP Based Control System for Free Electron Laser', Bulletin of the Polish Academy of Sci., Vol. 53, No. 2, pp. 123-138, 2005.
18.	J. Tratnik et al., 'Fiber Length Compensated Transmission of 2998.01 MHz RF Signal with Femtosecond Precision, Mic. Opt. Tech. Lett. Vol. 53, No. 7, pp.1553-1555, 2011.
19.	A. Ivanov et al., 'Progress on an Optical Link for Ultra-Stable Time Dissemination, Proceedings of PTTI2011, Long Beach (CA), Paper 10, 2011.
20.	Instrumentation Technologies, [http://www.i-tech.si/accelerators-instrumentation/libera-sync-fel/performance].
Arrived: 02. 04. 2012
Accepted: 30. 08. 2012