Skip to main content

Dome troubleshooting

Symptom-indexed. Work through the troubleshooting method first — in particular, establish whether the problem affects one channel or all channels, because that single fact splits this page in half.

TODO verify with senior technician

The symptoms below are derived from the dome's equipment documentation and from how the signal chain is built. The causes and step sequences are informed reconstructions and must be reviewed against real fault history before being treated as authoritative.

Quick triage​

ObservationMost likely layerJump to
One channel black, others fineDecoder or projectorOne projector channel is black
All channels black, projectors onPlayback or clusterAll channels black
Image doubled/ghosted in 3DPlayback modeStereo ghosting
Seams or misalignment visibleWarp/blend or node identityBlend and warp
Image tears between nodesGenlock/syncNodes out of sync
Decoder missing from status pageNetwork or powerEncoder/decoder link lost
Projector alarmingCoolingChiller alarm
Pre-show screens darkWATCHPAX4Pre-show not playing

One projector channel is black​

Likely causes: decoder lost its stream; projector shuttered, on the wrong input, or in standby; PDU outlet off; render node output failed.

  1. Open the Barco status page. Is the decoder for that channel present and linked?
  2. Open the projector's web interface — addresses on Projectors, http://172.19.7.3x:9999/. Check power state, shutter/dowser, selected input and any active alarms.
  3. If the projector is unreachable, check its PDU outlet on PDUs. Confirm the outlet name matches the projector before doing anything.
  4. If the projector is reachable and healthy but shows nothing, suspect the DisplayPort link between decoder and projector.

Do not power-cycle the projector if a session is imminent — a projector caught mid-cooldown may refuse to restart.

All channels are black, but the projectors are on​

This is almost never a projector problem. Work upwards from the content.

  1. Is the playback application still running on the render cluster? Has it crashed or dropped to the desktop?
  2. Is the correct source selected on the Showmaster tech panel?
  3. Are the render nodes still powered and logged in? See ATLAS / DRACO.
  4. Do the projectors show their own on-screen menu? If yes, everything from the projector down is fine and the fault is upstream.

Stereo (3D) ghosting or crosstalk​

Likely causes: playback is in the wrong stereo mode; content is not actually stereo; wrong glasses; eye channels swapped.

  1. Check the playback mode first. Mono content in a stereo show mode, or a left/right swap, produces exactly this symptom.
  2. Confirm the audience has Dolby 3D glasses — glasses from another venue or another 3D system will not separate the 6P laser wavelengths correctly.
  3. Verify the per-eye assignment on the render nodes. On the ATLAS cluster the stereo is emulated through dual DisplayPort outputs (i.e. passive stereo) — see ATLAS cluster, including the NVIDIA settings screenshots.
  4. Only if all of the above are correct should the projectors be suspected. That is an escalation, not a runtime fix.

Visible seams or misaligned blending​

Likely causes: a projector was physically knocked or serviced; a node is rendering the wrong frustum; calibration was not restored after a GPU driver change.

  1. Establish whether it is one seam (one projector moved or is rendering wrong) or all seams (a global calibration or content-format problem).
  2. Check that the affected node is rendering its correct channel — a node restored from a backup or renamed will happily render the wrong part of the dome.
  3. Check whether a GPU driver or display-configuration change happened recently. Warp and blend do not survive a display topology change.
Do not re-warp

Adjusting warp or blend is not a troubleshooting step. It is a calibration task with its own procedure and it can make the dome unusable if done wrong. Escalate.

Cluster nodes are out of sync​

Symptoms: tearing across a blend seam, one channel a frame behind, stutter that moves between channels.

  1. Confirm which cluster is in use and which node is the sync master. For ATLAS the sync master is ATLAS-01.
  2. Check the NVIDIA sync configuration on the master and on a sync node — reference screenshots are on ATLAS cluster.
  3. If you must restart nodes: restart the sync master last. The other nodes need something to lock to as they come back.
  4. Check display scaling and taskbar settings, which are part of the documented node configuration: scaling must be 100% on all nodes, and on ATLAS-01 to ATLAS-04 the taskbar sits at the top and auto-hides. A restored or reimaged node that differs here will render incorrectly.

The dome's video distribution is a Barco ENA-340 virtual KVM over 10 G fibre — see Barco ENA-340.

  1. Open the status page and note the Unit number of the missing device. Its IP is 172.19.7.<unit>, and the unit number links to the device web GUI.
  2. If the unit is completely absent, check its PDU outlet — dome decoders such as dome-ena-dec01 and dome-ena-dec02 are named on PDUs.
  3. If the unit is present but not passing video, check that it is bound to the correct stream — routing is done from the control computer.
  4. Check the fibre pair physically only after the above; SFP+ pairs do occasionally seat badly after rack work.

Encoders and decoders are interchangeable within the fleet. Record the unit number before swapping so the routing can be reapplied.

Chiller alarm or projector overheating​

The six XDL projectors each have a separate chiller, frequently in a different room — see the position table on the overview.

  1. Identify which chiller serves the alarming projector. Getting this wrong sends you to the wrong floor.
  2. Check the chiller for alarms, flow and coolant level.
  3. Check the chiller's power feed on PDUs.
  4. Do not keep restarting a projector that is thermally alarming. Escalate.

Pre-show content is not playing​

The four pre-show screens are driven by a Dataton WATCHPAX4 with WATCHOUT built in — see WATCHPAX4 pre-show.

  1. Is the WATCHPAX4 powered and on the network?
  2. Is the correct show loaded and running?
  3. Are the screens themselves powered and on the right input?
  4. Restart the WATCHPAX4 before restarting anything else — it drives nothing but the pre-show screens, so it is a low-risk restart.

Audio channel missing​

The dome runs a 7.2 system used as 7.1, through a DiGiCo mixer and Crown amplifiers.

  1. Determine whether the channel is missing at the source (content), at the mixer, or at the amplifier.
  2. Check mute and routing on the mixer before touching amplifiers.
  3. Check amplifier power and fault indicators.
TODO verify with senior technician

Add the speaker-to-amplifier-to-mixer-channel mapping and the location of the DiGiCo and Crown racks.

Lighting is not responding​

House lighting is an ETC Element 40 desk driving 142 Philips Color Kinetics fixtures plus four zoomed spotlights.

  1. Does the desk respond locally? If yes, the problem is in the DMX path or the fixtures.
  2. If the lighting is normally triggered from the Showmaster, test the desk directly to split control-system faults from lighting faults.
  3. Check the MIDI, relay and RS-232 gateways if the trigger path runs through one.

The Showmaster is unresponsive​

Showmaster 1 is the dome's controller. Three ways to restart it, least disruptive first — full detail on Showmaster.

  1. Reboot from the Showmaster itself — browse to showmaster1.c.itn.liu.se, open TechPanel, enter the password, and press Showmaster under the Reboot section. Completes in under a minute.
  2. Reboot via Medialon Showmaster Editor from another machine on the same network. For the dome this means VMIX-01 (172.19.6.0/24). Start the editor, accept the UAC prompt, choose Showmaster → Connect (Ctrl+Alt+K), select the unit, connect, then choose Reboot.
  3. Power cycle via ePDU — note that the tech panel's Power Cycle section is not yet implemented for our Showmasters, so this must be done from the PDU directly. Takes roughly two minutes.

Restart order for the dome​

When a broad restart is genuinely needed and no audience is present:

  1. Playback application
  2. Render nodes — sync master (ATLAS-01) last
  3. Decoders / encoders, if the status page shows them unhealthy
  4. Showmaster
  5. Projectors — only with full cooldown respected
  6. PDU outlets — last resort only

When to swap hardware​

DeviceSwappable?
ENA-340 encoder / decoderYes — record the unit number first
PDUYes — see spare units
Render nodeOnly if a spare is prepared; node identity and NVIDIA sync settings must be reapplied
ProjectorNo — warp and blend are per-position. Escalate.
ChillerEscalate