One key requirement when capturing Ethernet packets is knowing when the packet was captured. This is useful for a number of reasons including: network troubleshooting, network and application performance monitoring, network security analysis, compliance requirements for reporting e.g. the MiFID II regulations (RTS-25) and many more.
There are several different kinds of timestamping packet capture devices. Some pass frames through them with a timestamp and other per-frame metadata prepended or appended to the frame. Others capture the frame and pass it to software for analysis and/or write it to disk. The way timestamps are stored also varies, with some solutions requiring post-processing which references specific keyframes/marker frames after capture to extract the timestamp. There are naturally pros and cons to each solution.
Types of Ethernet capture devices
There are several classes of Ethernet capture devices which broadly fall into two categories:
- Devices that ingest Ethernet frames, timestamp them and then egress them; usually with some kind of timestamp and in some cases, per-packet metadata appended to the frame. These are known as tap aggregation devices, packet brokers or timestamping matrix switches. Metamako's nanosecond-resolution timestamping tap aggregation solution, MetaWatch, offers this functionality on a Metamako K-series platform.
- Devices that ingest Ethernet frames, timestamp them and then either pass them to local applications and/or write them to persisted storage, typically encoding the timestamp in a header prepended to each raw Ethernet frame in the file. The de facto standard application programming interface (API) and file format for this type of capture is pcap (packet capture). Solarflare's SolarCapture System is an example of an analytics/storage capture appliance.
Both classes of capture device usually work in unison, with network traffic being captured and aggregated by a tap aggregation device and then sent directly to an analytics/storage capture appliance. The key advantages of using a tap aggregation device to feed an analytics/storage capture appliance are that capture solutions, such as MetaWatch, include a Layer 1 Ethernet switch giving them the ability to both tap bidirectional Ethernet links with and also sink mirrored ports from network switches.
Tap aggregation devices also typically allow oversubscription of links being tapped as the majority of network traffic, on average, only consumes a fraction of the available bandwidth. Most tap aggregation devices, do this by buffering incoming packets when the instantaneous sum of the traffic on all the links being aggregated exceeds the bandwidth available on the egress port(s). This makes it possible to capture a much larger number of network links cost-effectively by reducing the number of analytics/storage capture appliances required (Further reading: Network Traffic Capture & Aggregation: Why buffer size is crucial).
Packet timestamp format and location
a. Appending a packet trailer
A number of tap aggregation devices exist on the market and handle timestamping packets somewhat differently. In the MetaWatch solution, the timestamp associated with each packet is formatted as a field in an industry-standard trailer containing the packet capture time as a UNIX time in seconds and nanoseconds as well as other useful fields such as a configurable identifier for the device and the port number that the packet came in on. This trailer is appended after the frame's original Ethernet frame check sequence (FCS) and a new, valid Ethernet FCS is generated immediately following the trailer.
The result is a valid Ethernet frame containing the original captured frame and its metadata in a self-contained format that can then be egressed from an Ethernet port, transition through Ethernet devices, if required, before being captured by an analytics/storage capture appliance. Wireshark, a free and open source network protocol analyser, is the de facto ad hoc tool for network troubleshooting and analysis and natively decodes the Metamako timestamp trailer format. A number of vendors offer similar self-contained metadata attached to the captured frame in either a header or trailer to the frame.
b. Appending/inserting a counter value
Unfortunately, the way that some tap aggregation devices store timestamps is not quite so straightforward. Rather than append a trailer containing a full, self-contained timestamp representing Unix time with nanosecond (or lower) resolution, along with other relevant metadata, they append a counter value. This counter value is typically a 32-bit field (or less) and depending upon configuration, it may replace the frame's Ethernet FCS. Doing so removes the ability to validate that the captured Ethernet frame has been captured uncorrupted. Once each frame has been captured by an analytics/storage appliance, this counter needs to be read and converted into a timestamp typically representing Unix time and its fractional component. To do so, two pieces of supplementary information are required. The first is the frequency of the counter which is nominally fixed, the second is usually provided by special Ethernet frames, known as keyframes or marker frames, generated by the tap aggregation device and injected into the capture stream, typically every second. The software converting each frame's appended counter field to a timestamp therefore needs to know precisely what trailer specification containing the counter value is being used in order to generate its actual full timestamp. Depending upon when capture started, the software needs to buffer frames for up to a second until the next keyframe/marker frame arrives and then use the information contained within it to generate a timestamp for each of the frames it has already received.
c. Approaches compared
Appending/inserting a counter value raises a few potential issues:
- A capture buffer, potentially gigabytes in size, needs to be maintained
- Enough processing power needs to be available to generate a timestamp for each frame individually in near real-time
- Receiving an aggregated stream containing frames from different timestamping domains can make identifying which keyframe/marker frame should be used to generate the timestamp difficult
None of the above are insurmountable, however, this approach to timestamping does add significant complexity in requiring a per frame decode referencing data obtained periodically from a keyframe/marker frame.
In contrast, the potential disadvantage of the typically longer self-contained trailers is that they do add to the frame size and hence buffers fill faster and hold fewer frames, particularly when average frame sizes are small. In situations where buffer size is at a premium, this is very much a valid concern. Tap aggregation solutions, such as MetaWatch, now exist with multiple gigabytes of buffer. The concern is still valid, however, in most practical scenarios, the buffers are deep enough that the impact of longer trailers is negligible.
Ethernet packet metadata other than the timestamp
Knowing definitively what specific port and device a particular packet was captured on is not just useful, it can be crucial. Take the case where a network has multiple capture points and the requirement is to track the progress of a packet through the network by comparing its timestamps at each capture point on its path. When each instance of the captured packet's trailers contains the port and device ID relating to its capture, it greatly facilitates identifying packets of interest without having to search through all packets captured on ports and devices not in the packet's path.
In the Metamako timestamp trailer, the first bit of the 8-bit Flags field indicates whether the original Ethernet FCS was valid. It is therefore possible to identify valid/invalid frames simply by reading this bit from each frame's trailer rather than having to recalculate and validate each frame's FCS.
Given the abundance of analysis/storage devices, their ability to parse and extract packet metadata is key to taking advantage of the benefits of it being present. A number of such devices, including Solarflare's SolarCapture System, the Corvil platforms and Velocimetrics' TipOff can extract the fields from the Metamako timestamp trailer and integrate them into their indexing and query interfaces.
- There are two main types of devices capturing and timestamping Ethernet packets:
- Tap aggregation devices/packet brokers/timestamping matrix switches which typically append a timestamp to each packet before forwarding it out of an Ethernet interface
- Analytics/storage devices which timestamp incoming packets and present them to applications via an API and/or write them to persisted storage, typically in pcap format
- These devices are often used to complement each other with tap aggregation devices timestamping, aggregating and buffering multiple links before sending them to analytics/storage devices
- Several different ways exist of appending timestamps and, in some cases, additional metadata to captured packets leaving tap aggregation devices:
- In the case of MetaWatch, an Ethernet packet trailer preserving the original FCS and terminating in a new FCS is used. This trailer contains a full Unix time timestamp with nanosecond resolution as well as key metadata including the port ID the packet came in on, the device ID and a flag indicating whether the original FCS was valid. Other vendors offer headers or trailers containing similar timestamps and per-packet metadata.
- An Ethernet packet trailer containing a counter value from which, when combined with the correct keyframe/marker frame, a Unix time timestamp may be generated. Additional per-frame metadata varies from none to comparable to that offered by vendors offering self-contained metadata
- Both approaches have their pros and cons however given than tap aggregation devices such as MetaWatch offer multiple gigabytes of buffering, the benefits of self-contained timestamps outweighs the associated loss of buffer capacity in most practical scenarios