Optical Interconnects at the Chip and
Board Level – Challenges and Solutions
David V. Plant and Andrew G. Kirk
McGill University
Montreal, Quebec, Canada
(plant@photonics.ece.mcgill.ca)
May 15, 2003, Stanford University
Acknowledgements:
McGill Photonic Systems Group
Dr. Mani Sundaram, TeraConnect
CITR
D.V. Plant
1
System interconnect hierarchy
Inter-rack
Network
Element
Intershelf
Inter-site
Network
Elements
Long-haul
Inter-board
Inter-chip
D.V. Plant
Central office
2
Outline
¾Bandwidth limitations of electrical
interconnects
¾Optical interconnect insertion points
¾10 Gbps products
¾Free-space optical interconnects
¾Convergence/divergence
¾Conclusions
D.V. Plant
3
Problem statement
• Electrical interconnects do not have sufficient
bandwidth to meet the interconnection requirements of
future telecommunications and computing systems.
• Short distance interconnects will benefit from an
optical solution.
• Multiple groups pursuing solutions:
– Telecommunications equipment manufacturers
– Computing systems manufacturers
– Storage/disk farms
D.V. Plant
4
Limitations of electrical interconnects
(D.A.B.Miller and H.M.Ozaktas, J.Parallel Distrib. Comput., 41, pp 42-52,
1997)
• As data rates increase, electrical interconnects are limited by:
– Power
– Distortion
– Cross-talk
– Pin-out capacity
• Fundamental: aspect ratio limit:
Bmax ≤ 1016 ×
D.V. Plant
A
l2
l
A
(bit/s )
5
Opportunities for optical
interconnects
Longhaul
LAN/WAN
Racks/chassis
VSR optical
interconnects
1mm
1cm
10cm
1m
10m
100m
1km
CWDM and TDM:
λ, time muxing
(single mode fiber,
multimode fiber)
D.V. Plant
10km
100km
1000km
DWDM:
λ, time muxing
(single mode
fiber)
6
VSR application: intra-office
(D. Goodwill, Nortel Networks)
• 2 to 300 meter interconnect
• 10 Gbps
D.V. Plant
7
VSR optical interconnect technology
requirements
9Interconnect opaque elements
9Large aggregate data rates (10, 40, 100 Gpbs)
9Multiple distance possibilities (2m to 10 km)
9Low power (< 3 W/10 Gbps)
9Standardized interfaces (electrical and optical)
9Low cost (~$100/Gbps)
D.V. Plant
8
Enabling VSR technologies
• Vertical Cavity Surface Emitting Lasers
(VCSELs)
• Fiber (single/multi-mode)
• Connectorized optical interfaces
• Standardized electrical interfaces
D.V. Plant
9
VCSELs
• Optical cavity oriented normal to
wafer.
• Current injection is efficient, slope
efficiencies are high, and threshold
currents are very low (sub 100 µA
reported in late 1990s).
• AlGaAs-GaAs (850 nm) VCSELs
are available at low cost.
• Current research on 1.3µm and
1.55µm devices.
D.V. Plant
10
VCSEL merits
¾ Inexpensive (Honeywell 850nm VCSEL $10)
¾ Bandwidths of 10Gbps at threshold currents of mA’s
¾ Wafer testable prior to doing any packaging
¾ Post-processing of mirrors not required
¾ Two-dimensional arrays available
1-D array
2-D array
D.V. Plant
11
Fiber types
Single mode fiber
Distance r
Multimode fiber (GRIN)
Distance r
Core diameter
Core diameter
0
0
Refractive
index n
D.V. Plant
• Step index core
• Core diameter 8-10 µm
• Bandwidth-length
product limited by
chromatic dispersion
• Reach at 10 Gbps > 10’s
of km
Refractive
index n
• Gradient refractive index
(GRIN) core
• Core diameter 50/62.5 µm
• Bandwidth-length product
limited by intermodal
dispersion
• Reach at 10 Gbps ≈ 60 m at
850 nm
12
VSR driver: 10 Gig Ethernet
standards
(http://grouper.ieee.org/groups/802/3/ae/index.html)
• 3 Serial:
– 850 nm, 2m – 300m on high performance multi-mode fiber
– 1310 nm, 2m – 10km on standard single mode fiber
– 1550 nm, 2m – 40km on standard single mode fiber
• 1 Parallel:
– 4 wavelengths in the 1310 nm window (1269nm – 1355nm);
2m – 300m on MMF; 2m – 10km on SMF
• Current products use 850nm VCSELs and 1310nm
DFBs; 1310nm VCSELs coming on fast.
D.V. Plant
13
VCSEL based 10 Gbps optical module
(D. Kabal and M. Ayliffe, Picolight)
• Applications:
– 10 Gigabit Ethernet (WAN and
LAN)
– OIF VSR OC-192
– Router interconnection
– Computer cluster cross
connects
• Low power: < 6 W
• 0 to 70 C
• 2 – 300 meters
• 50/125 µm MMF
D.V. Plant
14
10.7 Gpbs 850 nm VCSEL
(D. Kabal and M. Ayliffe, Picolight)
D.V. Plant
15
Data rate (Gb/s)
Current products: channels vs. data
rate
10000
10000
1000
1000
100
100
40 Gbps to follow
10
10
1
10 Gbps
1
D.V. Plant
1
10
100
1000
Number of channels
10000
16
System interconnects
Inter-rack
Network
Element
Intershelf
D.V. Plant
Network
Elements
Long-haul
Inter-board
Inter-chip
Inter-site
“Inside
the box”
Central office
17
Opportunities for optical
interconnects
Longhaul
LAN/WAN
Racks/chassis
Inter-shelf
2-D VSR optical
interconnects
Inter- Board
Inter- Chip
1mm
1cm
10cm
1m
2D SDM & TDM:
space, time muxing
(FSOI, parallel fiber)
D.V. Plant
10m
100m
1km
CWDM & TDM:
λ, time muxing
(single mode fiber,
multimode fiber)
10km
100km
1000km
DWDM:
λ, time muxing
(single mode
fiber)
18
2-D Free Space Optical Interconnects
OE-VLSI
ASIC
OE devices
(>2000/cm2)
Optics
• Free Space Optical Interconnect (FSOI) bandwidths achieved through
Time Division Multiplexing (TDM) and Space Division Multiplexing (SDM).
• Connection densities >2000 connections/cm2 at > 1Gbps per connection.
• OE-VLSI ASIC baseline functions: Electrical-to-Optical (E-O); Optical-toElectrical (O-E); Optical-to-Electrical-to-Optical (O-E-O) data conversion.
D.V. Plant
19
Application: free-space optical
backplane
D.V. Plant
20
Application: chip-to-chip/board-toboard
(1999 edition, http://public.itrs.net)
International Technology Roadmap for Semiconductors
Year of Introduction
Technology node
1999
180 nm
2003
120 nm
2005
100 nm
2008
70 nm
2011
50 nm
2014
35 nm
Density (M/cm2)
24
78
142
350
863
2,130
Chip size (mm2)
450
567
662
713
817
937
1,250
2,490
3,500
6,000
10,000
13,500
Off-chip high-speed pins
700
1,500
1,900
2,300
2,700
3,000
Off-chip speed (MHz)
600
862
1,000
1,250
1,500
1,800
On-chip clock (MHz)
(Data for high-performance market segment)
D.V. Plant
21
Application: chip-to-chip/board-toboard
(1999 edition, http://public.itrs.net)
International Technology Roadmap for Semiconductors
Year of Introduction
Technology node
1999
180 nm
2003
120 nm
2005
100 nm
2008
70 nm
2011
50 nm
2014
35 nm
Density (M/cm2)
24
78
142
350
863
2,130
Chip size (mm2)
450
567
662
713
817
937
1,250
2,490
3,500
6,000
10,000
13,500
Off-chip high-speed pins
700
1,500
1,900
2,300
2,700
3,000
Off-chip speed (MHz)
600
862
1,000
1,250
1,500
1,800
On-chip clock (MHz)
‘No known solution’
D.V. Plant
22
Data rate (Gb/s)
Chip-to-chip requirements
10000
10000
1000
1000
5 Tbps (2014)
100
40 Gbps
10
1
On-chip clock
(13.5 Gbps)
Off-chip bus
(1.8 Gbps)
10 Gbps
1
D.V. Plant
100
10
1
10
100
1000
Number of channels
10000
23
2-D VSR optics: open issues
• Total interconnect capacity
• Number of channels/data rate per channel
• Length
• Optoelectronics
• Optics
D.V. Plant
24
Starting Assumptions
• 2D VCSELs and
photodiodes arrays @
850 nm, mW’s.
PCB
Optoelectronic receiver array
• OED Pitch: 125 µm
• CMOS, SiGe, UTSi
electronics
• Interconnect distances:
5 – 100 mm
• FSOI optical
interconnect insertion
loss < 3 dB
D.V. Plant
Optical
interconnect
Optoelectronic transmitter array
VLSI circuit
Low speed electrical connections
25
Optoelectronic-VLSI
Multichip TXER module
CMOS chip
Driver VCSEL
Array
Heterogeneous integration
OE-VLSI
Increased parallelism
Reduced power dissipation
D.V. Plant
26
ASIC #1) 256 VCSEL/PD OE-VLSI ASIC
(D.V. Plant et al, JLT, 19, 2001)
10mm x 10 mm, 0.35 µm TSMC CMOS
D.V. Plant
27
Designed as Network Interface Chip
(a)
(b)
D.V. Plant
OEO operation at a) 250 Mbps; b) 400 Mbps.
• Pitch within
cluster:
125µm
• Cluster
pitch: 750
µm
• 2844
channels/cm
2
• Performed 3
functions: EO, O-E, and
O-E-O
28
ASIC #2) 1080 VCSEL/PD OE-VLSI ASIC
(M.B. Venditti et al, in press, JSTQE, 2003)
Receiver
Array
Digital
Section
Transmitter
Array
14.6 x 7.5 mm 0.25 µm TSMC CMOS (5 metal, single poly, n-well)
D.V. Plant
29
Signal Processing Functions
• Transceiver/
transponder
• Clock
acceleration
• Parallel Forward
Error Correction
(FEC)
• On chip data
generation
(testing)
D.V. Plant
30
Test and Verification
Toggle
Blink
Random
Close-up
D.V. Plant
31
OE-VLSI ASIC Comparison
ASIC #1
ASIC #2
Optical I/O
256 (single ended)
540 (differential)
Electrical I/O
32 (single ended)
128 (single ended)
Transceivers
Single ended
Fully differential
Total power dissipation
(calculated)
~ 100 mW/ch
~ 35 mW/ch
Connection density
2844 chs/cm2
6800 chs/cm2
Number of functions
Transceiver &
transponder
Transceiver/transponder;
clock acceleration, FEC,
self-test
D.V. Plant
32
Optically Interconnected PCBs
OE-VLSI
ASIC
~ 1 inch
separation
• Backplane OE-VLSI ASICs perform:
– E-O, O-E, and O-E-O data conversion
– Backplane level signal processing functions such as a) address
recognition, b) clock acceleration, c) FEC
D.V. Plant
33
Chip-to-Chip Interconnect
3 inch
separation
• Forms basis of technologies to be discussed:
– VCSELs/PDs & heterogeneous integration
– OE-VLSI ASICS (transceivers, layout, signal processing)
– Packaging and optics
D.V. Plant
34
2-D VSR optical interconnects: design
choices
• Critical design parameters:
Throw
– Optical throw
– Number of channels
– Array area
– Misalignment tolerance
– Source characteristics
Area
Misalignment
tolerance
• These cannot all be
optimized independently
D.V. Plant
35
Optical relay choices
Microchannel
• Short throw (diffraction limited)
• Simple lenses
• Tight lateral tolerances
Clustered
(minilens)
• Longer throw
• More complex optical design
• Improved lateral tolerances
Macrolens
• Maximum throw
• Stresses field of view
• Aberration-limited
D.V. Plant
36
Optical relay choices
Clustered
(minilens)
• Longer throw
• More complex optical design
• Improved lateral tolerances
D.V. Plant
37
Design for misalignment tolerance
• Misalignment results from manufacturing and assembly errors
• Systems designed to be tolerant to misalignment
• Calculate the impact of misalignment on performance
8-f optical relay
D.V. Plant
38
Design for misalignment tolerance
• Misalignment results from manufacturing errors, assembly
• Systems should be tolerant to misalignment
• Calculate the impact of misalignment on performance
• What happens when all components are misaligned ?
ÆA Monte-Carlo simulation is required
8-f optical relay
D.V. Plant
39
Modularization
• Typically systems have components that have critical
alignment tolerances
• This requires that the system is assembled as a series
of modules
• Each module is comprised of components that must be
mutually aligned with care – tight tolerance
• Ideally the modules can then be assembled with
passive mechanical alignment – loose tolerance
D.V. Plant
40
Example: 512 channel optical board
to board link 12.5 µm tolerance
Minilens
Relay system
Prism
IC
VCSEL
cluster
D.V. Plant
Prism
Glass spacer
Microlens array
Detector
cluster
•
•
•
•
86 mm throw
3x6 mm active area
256 channels (bidirectional)
Æ Clustered optical design
2.5 µm tolerance
(M. Châteauneuf et al, Optics in Computing 2001, pp.64-66.)
41
512 channel parallel optical board to
board link
Minilens
Relay system
Prism
IC
VCSEL
cluster
D.V. Plant
Prism
Glass spacer
Microlens array
Detector
cluster
•
•
•
•
86 mm throw
3x6 mm active area
256 channels (bidirectional)
Æ Clustered optical design
90 µm tolerance
(M. Châteauneuf et al, Optics in Computing 2001, pp.64-66.)
42
Assembly and packaging
• Module assembly techniques include:
– Interferometric techniques
– In situ alignment techniques
– Active alignment
• Module integration requires molded or machined
features
D.V. Plant
43
Packaging challenge and
solutions
Machined frame
Alignment lenses
Ceramic spacers
Relay lenses
Mechanical
alignment features
D.V. Plant
Microlens array
Relay block
44
Completed 256 channel bidirectional
system*
D.V. Plant
*M. Châteauneuf et al, Optics in Computing 2001, pp.64-66.
45
Opportunities for optical
interconnects
Longhaul
LAN/WAN
Racks/chassis
Inter-shelf
1D, 2D VSR optical
interconnects
Inter- Board
Inter- Chip
1mm
1cm
10cm
1m
2D SDM & TDM:
space, time muxing
(FSOI, parallel fiber)
D.V. Plant
10m
100m
1km
CWDM & TDM:
λ, time muxing
(single mode fiber,
multimode fiber)
10km
100km
1000km
DWDM:
λ, time muxing
(single mode
fiber)
46
VSR optical interconnect
requirements
10000
10000
1,2-D Fiber
2-D FSOI
1000
Data rate (Gb/s)
1000
5 Tbps
100
On-chip clock
(13.5 Gbps)
40 Gbps
10
Xanoptix
10 Gbps
Off-chip bus
(1.8 Gbps)
Teraconnect
1
2-D fiber
1
D.V. Plant
100
10
10
1
McGill/CITR
100
1000
Number of channels
10000
47
Convergent challenges
¾VCSELs with low threshold currents (Ith)
¾Low loss optical interconnect
¾Low power consumption
¾Non-amplified link
¾Low cost
D.V. Plant
48
Divergent challenges
Parameter
Data rate per
channel
Length
Telecomm
10 – 100 Gbps
Computing
1 – 10 Gbps
Interconnect
topology
Meters to
kilometers
Moderate & 1D
(100’s connections)
Optical medium
Fiber (SM/MM)
Connectorization
Electrical interface
Yes
Serial/de-serial
Millimeters to
meters
High and 2D (100’s
to 1000’s of
connections)
2-D; FSOIs, parallel
fiber
Desirable
Directly from ASIC
D.V. Plant
49
Conclusion
Inter-rack
Network
Element
Intershelf
Inter-board
D.V. Plant
Network
Elements
Long-haul
Mature
(10 GigE)
Inter-chip
Inter-site
Research
Very
mature
Central office
• Optical interconnects continue to penetrate systems
• New solutions for shorter reaches required
50
References
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11)
12)
13)
D.V. Plant
Kirk, A.G., D.V. Plant, M. Ayliffe, M. Chateauneuf, and F. Lacroix. Design rules for highly parallel free-space optical interconnects. In
press, IEEE Journal of Selected Topics in Quantum Electronics, (2003) [invited].
Venditti, M.B. and D.V. Plant. On the design of large receiver and transmitter arrays for OE-VLSI applications. Submitted to IEEE/OSA
Journal of Lightwave Technology, December (2002).
Venditti, M.B., E. Laprise, J. Faucher, P-O. Laprise, J.E. Lugo, and D.V. Plant. Design and test of an optoelectronic-VLSI chip with 540element receiver/transmitter arrays using differential signaling. In press, IEEE Journal of Selected Topics in Quantum Electronics,
(2003).
Kirk, A.G., D.V. Plant, T.H. Szymanski, Z.G. Vranesic, F.A.P. Tooley, D.R. Rolston, M.H. Ayliffe, F. Lacroix B. Robertson, E. Bernier,
and D. F.-Brosseau. Design and implementaion of a modulator based free-space optical backplane for microprocessor applications. In
press, Applied Optics, (2003).
Faucher, J., M.B. Venditti, and D.V. Plant. Design and implementation of forward error correction for 2-dimensional parallel optical data
links (2D-ODLs). IEEE/OSA Journal of Lightwave Technology, in press (2003).
Chateauneuf, M., A.G. Kirk, D.V. Plant, T. Yamamoto, J.S. Ahearn, and W. Luo. 512-channel vertical cavity surface emitting laser
based free-space optical link. Applied Optics 41: 5552-55561 (2002).
Lacroix, F., E. Bernier, M.H. Ayliffe, F.A.P. Tooley, D.V. Plant, and A.G. Kirk. Implementation of a compact, four stage, scalable
optical interconnect for photonic backplane applications. Applied Optics 41: 1541-1555 (2002).
Ayliffe, M.H., M. Chateauneuf, D.R. Rolston, A.G. Kirk, and D.V. Plant. Six-degrees-of-freedom alignment of two-dimensional array
components using in-situ off-axis diffractive structures. Applied Optics 40: 6515-6526 (2001).
Plant, D.V., M.B. Venditti, E. Laprise, J. Faucher, K. Razavi, M. Chateauneuf, A.G. Kirk, and J.S. Ahearn. 256-channel bidirectional
optical interconnect using VCSELs and photodiodes on CMOS. IEEE/OSA Journal of Lightwave Technology 19: 1093-1103 (2001).
Ayliffe, M.H., D.R. Rolston, A. Chuah, E. Bernier, F.S.J. Michael, D. Kabal, A.G. Kirk, and D.V. Plant. Design and testing of a
kinematic package supporting a 32x32 array of GaAs MQWs flip-chip bonded to a CMOA chip. IEEE/OSA Journal of Lightwave
Technology 19: 1543-1559 (2001).
Venditti, M.B., E. Laprise, J. Faucher, P.-O. Laprise, J. S. Ahearn, and D. V. Plant. Design and verification of an OE-VLSI chip with
1080 VCSELs and PDs heterogeneously integrated with CMOS. Proc. IEEE/LEOS 2001 Annual Meeting, PD-1.4. (2001).
Plant, D.V. and A.G. Kirk. Optical interconnects at the board and chip level: Challenges and solutions. Proceedings of the IEEE 88: 8896 (2000) [invited].
Maj, T., A.G. Kirk, D.V. Plant, J.F. Ahadian, C.G. Fonstad, K.L. Lear, K. Tatah, M.W. Robinson, and J.A. Trezza. Interconnection of a
two-dimensional array of Vertical Cavity Surface Emitting Lasers (VCSELs) to a receiver array by means of a fiber image guide. Applied
Optics 39: 685-689 (2000).
51