This reference architecture uses ConnectX-6 Dx adapters on two independent Ethernet fabrics for market-data ingestion, order processing, risk services, and deterministic time measurement. The design prioritizes bounded tail latency, loss visibility, clock accuracy, and predictable recovery rather than maximum headline bandwidth. It is not a claim of a specific microsecond improvement or trading advantage. Results depend on exchange connectivity, application code, CPU and NUMA placement, switch behavior, operating-system tuning, packet rates, and test conditions.
A low-latency trading network must be evaluated as a complete path. The NIC can accelerate packet processing and provide precise timestamps, but queueing can still occur in an application, kernel, switch, gateway, or risk service. Optimizing one median while leaving long p99.9 outliers unmeasured can make the platform look faster without making it more predictable.
Reference architecture at a glance
| Layer | Reference design | Primary validation |
|---|---|---|
| External connectivity | Diverse carrier or exchange paths with separately documented demarcation | Path diversity, packet sequence, loss, and recovery time |
| Fabric | Independent A and B Ethernet switches and cable routes | No shared failure domain, consistent policy, and deterministic failover |
| Server adapter | ConnectX-6 Dx ports assigned to explicit NUMA and application roles | PCIe width, interrupt or polling placement, timestamp accuracy, and counters |
| Host software | Controlled firmware, driver, kernel-bypass or socket path, CPU isolation, and memory policy | Repeatable configuration and latency distribution under burst |
| Timing | Redundant approved time sources distributed to hosts and network devices | Offset, holdover, source change, timestamp correlation, and alerting |
Why ConnectX-6 Dx remains relevant
NVIDIA specifies ConnectX-6 Dx with up to two 25, 50, or 100Gb/s ports or one 200Gb/s Ethernet port and PCIe Gen4 host connectivity. The family includes hardware offloads, RoCE support, security acceleration, and accurate time-synchronization services. These capabilities fit environments where packet rate, latency measurement, and host efficiency matter more than moving to a 400G generation.
The exact orderable part still determines ports, cage, form factor, crypto capability, bracket, and host compatibility. One site example is the ConnectX-6 Dx MCX623105AN-VDAT. Validate it against the server and intended optics; do not infer that all ConnectX-6 Dx models have the same connector or feature set.
Use two fabrics as independent systems
Fabric A and Fabric B should not be two VLANs on the same switch. Separate switches, power feeds, cable routes, management addresses, and upstream paths allow the application to survive a planned component failure. Where the exchange or carrier offers diverse handoffs, map each to a defined internal fabric and verify where the paths converge outside the data center.
Application behavior during duplication and failover must be explicit. Some systems consume both feeds and remove duplicates; others prefer one path and switch after a health decision. Network redundancy does not decide which message is authoritative or how sequence gaps are handled. That logic belongs in a tested market-data and order-routing design.
Precision timing needs an end-to-end clock design
NVIDIA states that ConnectX NICs provide accurate time-synchronization services starting with ConnectX-6 Dx. Hardware timestamps can help separate network transit from host and application processing, but only when clocks are synchronized and every timestamp has a documented reference point.
Define the grandmaster or approved external source, redundant distribution, PTP profile, domain, boundary or transparent clock roles where used, holdover expectations, and alert thresholds. Test loss of the primary source and confirm how timestamps are marked during degraded synchronization. The audit trail should show clock offset and source state alongside order and market-data events.
Host topology can dominate the result
Place the NIC, application cores, and memory on the intended NUMA node. Check the server's actual PCIe tree and electrical width. Pin polling threads or interrupts according to the chosen software path, reserve CPU resources from unrelated workloads, control power-management settings, and document hyper-threading policy. Make one change at a time and retain the baseline.
Kernel bypass and DPDK can reduce host overhead for suitable applications, while conventional sockets may be operationally simpler and sufficient for other services. The architecture can use both, but their latency and observability characteristics should not be mixed in one result. Measure the production code path with production-equivalent message parsing and risk checks.
Keep switch policy small and inspectable
Use a switch platform and software release validated for the required link rates, optics, packet sizes, multicast behavior, access controls, telemetry, and timing. Minimize features on the latency-sensitive path. Every queue, policer, mirror, or security function must have a known reason and test case.
Low buffer occupancy can reduce queueing, but insufficient capacity can turn bursts into loss. The correct balance depends on traffic and recovery behavior. Capture microbursts, drops, pause events where applicable, and queue occupancy. A port with low average utilization can still be the source of tail-latency events.
Distinguish market data, orders, and bulk traffic
Market data may involve high packet rates, multicast, bursts, and sequence sensitivity. Orders and acknowledgements are usually lower bandwidth but have strict handling and audit requirements. Reference data, model distribution, logs, backup, and analytics are less latency-sensitive and should not share an unconstrained queue with the critical path.
Use separate physical interfaces, virtual routing, VLANs, or controlled traffic classes according to the assurance model. RoCE may be appropriate for specific internal analytics or storage workloads, but it should not be enabled on the trading path without a clear application requirement and complete congestion design.
Cabling and optics are part of latency operations
Select direct-attach copper, active cable, or optics by reach, cage, qualification, serviceability, and environmental constraints. Short copper links can reduce component count, while structured fiber may be required for diverse paths or longer routes. Keep A and B fabrics visually and physically identifiable without relying only on software labels.
Document adapter port, switch port, transceiver or cable part number, route, length, and timing role. Monitor error counters and optical levels where available. The Mellanox cable range should be filtered by the exact adapter and switch pair.
Validation must report distributions
- Measure one-way and round-trip latency with synchronized clocks and defined timestamp points.
- Report median, p95, p99, p99.9, maximum, packet loss, and sequence gaps, not a single best result.
- Replay normal, opening-burst, news-event, and recovery traffic with realistic message sizes and application processing.
- Run tests while monitoring CPU frequency, NUMA traffic, PCIe counters, NIC counters, switch queues, and clock offset.
- Remove a link, switch, time source, and application instance separately and measure service behavior.
- Repeat after firmware, driver, BIOS, kernel, or application changes that affect the path.
When ConnectX-7 may be the better choice
ConnectX-7 may be appropriate when a new platform requires 400G connectivity, PCIe Gen5 host bandwidth, higher port density, or a broader consolidation strategy. That does not make it inherently lower latency for every trading workload. Compare the exact adapter and server combinations under the real application path. The ConnectX-7 vs ConnectX-6 Dx guide covers the generation and compatibility boundaries.
Questions trading infrastructure teams should answer
Does a 200G NIC reduce order latency?
Not by itself. Higher link capacity can reduce congestion in a loaded path, but application processing, CPU placement, packet size, switch queueing, and upstream connectivity may dominate. Measure the complete path.
Can both redundant ports be bonded?
Bonding may improve availability for some services, but it can hide path identity and does not implement market-data deduplication or order-session state. Use the redundancy model required by the application and upstream venue.
Is hardware timestamping enough for regulatory records?
No. It is one component of a timing system. Clock traceability, synchronization state, timestamp point, retention, monitoring, and documented procedures must meet the applicable requirements.
Reference conclusion: a low-latency trading network is credible when its paths, clocks, host topology, queue behavior, and failure modes are measurable. ConnectX-6 Dx provides a capable Ethernet and precision-timing foundation, while disciplined system design determines repeatability and operational evidence.