Home / Blog / SIEM and Tools

Beginner SIEM and Tools

Time Sync and NTP: Why Wrong Clocks Break SOC Investigations

A beginner guide to NTP, stratum and clock drift, with Windows and Linux checks, event 4616, and a Splunk search that shows when log timestamps cannot be trusted.

Every log line carries a timestamp, and your whole investigation depends on those timestamps being right. This guide explains NTP (Network Time Protocol), how it keeps clocks in step, and what goes wrong in a SOC (Security Operations Centre) when it does not.

TopicTime synchronisation and NTP
You will meet it inWindows domains, Linux servers, firewalls, SIEM searches
Syllabus linkCySA+ CS0-003, objective 1.1
>

Scenario: The Windows event log on WS-114 shows a new service installed at 09:12. The firewall log shows WS-114 contacting 203.0.113.50 at 09:50. Did the connection come 38 minutes after the install, or did it happen first? Before you answer, you need to know whether those two clocks agree.

What time synchronisation means

Every computer, firewall and server keeps its own clock. Those clocks are never perfect, and small errors build up until two devices disagree by seconds, then by minutes. This slow wandering is called clock drift.

Time synchronisation fixes drift by making each device check its clock against a trusted source on a regular schedule. The standard way to do it is NTP. It runs over UDP (User Datagram Protocol) port 123, and the current version, NTPv4, is described in RFC 5905. An RFC (Request for Comments) is a numbered document that defines an internet standard.

One detail matters for analysts: NTP carries the time only. It does not tell a device which time zone or daylight saving rules to use. That is why many teams store and search logs in UTC (Coordinated Universal Time) and convert to local time only when they write a report.

CorrelationJoining events from the firewall, proxy and endpoint only works if their clocks agree.
AuthenticationKerberos in Windows domains rejects requests when two clocks are too far apart.
EvidenceA timeline in an incident report is only as trustworthy as its timestamps.

How time reaches your machines

The stratum idea

NTP organises time sources into levels called strata. Stratum 0 is the reference clock itself, such as a GPS receiver or an atomic clock. A stratum 1 server is connected directly to one of those. A server that gets its time from a stratum 1 server becomes stratum 2, and so on down the chain. RFC 5905 uses stratum 16 to mean "unsynchronised", so a device reporting 16 has no usable time source.

Stratum 0Reference clockGPS or atomicStratum 1Time serverfed by the clockStratum 2Internal serveror domain controllerStratum 3WorkstationWS-114
Each step away from the reference clock adds one stratum. Real networks vary, so the exact numbers will differ in your organisation.

Windows domains and the five minute rule

In an Active Directory domain, domain members normally get their time from the domain hierarchy instead of from a public server. Microsoft documents the domain controller that holds the PDC (Primary Domain Controller) emulator role as the authoritative time source for its domain, and the one in the forest root domain is the one that should take its time from an external source. Other domain controllers, workstations and member servers follow the chain down.

This matters because of Kerberos, the authentication protocol Active Directory uses. Kerberos puts timestamps in its messages to stop replay attacks, where an attacker resends a captured login message. Windows therefore refuses requests when the client and domain controller clocks differ by more than the policy "Maximum tolerance for computer clock synchronization", which defaults to 5 minutes.

When that happens, the domain controller can log Security event 4771 (Kerberos pre-authentication failed) with failure code 0x25, which means "clock skew too great".

!

Watch out: a burst of 4771 failures looks a lot like password guessing. Check the failure code first. Code 0x25 points to a clock problem, and a wrong password gives a different code.

Check a clock in two minutes

You do not need special tools. Run these on a lab machine first so you know what normal output looks like. On Windows, use an elevated Command Prompt (Run as administrator).

Windows: w32tm

Command Promptw32tm
w32tm /query /source
w32tm /query /status
w32tm /query /configuration

The first line shows where this machine gets its time. On a domain member you expect a domain controller. A source such as "Local CMOS Clock" or "Free-running System Clock" means it is not following a network time source. The second line shows the service status and timing details. The third shows the effective settings, including any that Group Policy supplies.

Linux: timedatectl and chrony

ShellBash
timedatectl
chronyc tracking

On systemd based systems, timedatectl prints the local time, UTC time, and whether the system clock is synchronised and the NTP service is active. If the machine runs chrony, chronyc tracking shows its reference source, its stratum, and a leap status that reads Normal when things are healthy. Which time service a Linux machine uses depends on the distribution, for example chrony or systemd-timesyncd, so check what is installed.

+

Tip: if a server never synchronises, check that outbound UDP port 123 to its time source is allowed through the firewall. It is one of the first things to rule out.

Back to the scenario. Running w32tm /query /status on WS-114 would show whether it is following a good source. If its clock is 38 minutes slow, the install at 09:12 on the workstation really happened at 09:50 on the firewall's clock. The two events then line up as the same moment, and the connection looks like it came straight from the new service.

When someone changes the clock on purpose

Sometimes the clock is not drifting, it is being changed. Windows records this as Security event 4616, "The system time was changed". The event lists the account that made the change, the process involved, the previous time and the new time. Normal corrections by the Windows Time service show the account as LOCAL SERVICE. Microsoft's guidance is to report any 4616 where the subject is not LOCAL SERVICE.

A changed clock can disrupt Kerberos and can shuffle a timeline. Microsoft's Sentinel hunting query for 4616 maps it to MITRE ATT&CK technique T1070 (Indicator Removal). Do not confuse that with T1070.006, Timestomp, which is about changing the time attributes of files rather than the system clock.

Microsoft SentinelKQL
SecurityEvent
| where TimeGenerated > ago(7d)
| where EventID == 4616
| where SubjectUserSid != "S-1-5-19"
| project TimeGenerated, Computer, SubjectAccount, ProcessName, PreviousTime, NewTime
| sort by TimeGenerated desc

Line 1 chooses the SecurityEvent table, which holds Windows Security events. Line 2 keeps the last seven days. Line 3 keeps only event 4616. Line 4 removes changes made by the built in Local Service account, whose well known SID (security identifier) is S-1-5-19, so routine Windows Time service corrections drop out. Line 5 picks the useful columns, and line 6 shows the newest first.

i

Note: Microsoft's own hunting query for this event says 4616 is only available when full event collection is enabled. If the search returns nothing, check what your connector collects before you decide nothing happened. Virtual machine tools, such as VMware Tools, can also change a guest clock legitimately, so a hit is a question to ask, not a verdict.

When you see a hit, look at three things: how big the jump was, whether it went backwards, and whether it happened outside normal working hours. Then ask the owner of the host what changed.

Spot bad timestamps in your SIEM

A SIEM (Security Information and Event Management) tool stores two times for many events: the time written in the log, and the time the SIEM received it. In Splunk these are the fields _time and _indextime. Comparing them gives a quick health check. The index and host names below are placeholders.

SplunkSPL
index=wineventlog earliest=-1h latest=now
| eval lag_seconds = _indextime - _time
| stats min(lag_seconds) as min_lag avg(lag_seconds) as avg_lag max(lag_seconds) as max_lag by host
| sort - max_lag

Line 1 searches one index for the last hour. Line 2 works out the gap in seconds between when each event was indexed and the time written in the event. Line 3 finds the smallest, average and largest gap for each host. Line 4 puts the biggest gap first.

Splunk's community guidance treats this difference as a rough measure of indexing delay. A gap of a few seconds is normal. A host with a huge gap is worth a look. A negative gap means the event claims to be from after the moment Splunk received it, which suggests that host's clock is ahead. A large positive gap can mean a slow clock, or logs that arrive in batches, so use it as a clue and confirm with the checks above.

Problem 1

Drift

Seconds or minutes of slow error because a device has no working time source.

Fix: restore its NTP source
Problem 2

Offset

A large steady gap, often caused by a wrong time zone setting or a bad time source.

Fix: correct the setting, log in UTC
Problem 3

Tampering

The clock was changed on purpose, shown by event 4616 from an unexpected account.

Action: escalate with the evidence

When two sources do not line up, work through a short routine before you write anything in your timeline.

  1. L0
    Pick one referenceChoose a source you trust, such as a domain controller or the SIEM itself.
    Anchor
  2. L1
    Compare each sourceFind an event you can see in both places and note the time each one shows.
    Compare
  3. L2
    Work out the offsetSubtract one time from the other and write down the size and direction.
    Measure
  4. L3
    Record and reportPut the offset in your notes and flag the broken clock to the owner.
    Escalate
  • Write every time in UTC in your notes, and say so.
  • Never merge two timelines until you have checked their clocks.
  • Treat 4771 with code 0x25 as a clock problem first.
  • Treat 4616 from any account other than LOCAL SERVICE as worth a question.
  • Remember that a clock fault is itself a finding, so tell the system owner.

What to study next

CompTIA's CySA+ (CS0-003) exam objectives list log ingestion, time synchronisation and logging levels together under objective 1.1 in the Security Operations domain. Exam versions change, so check which version you are sitting on CompTIA's website.

  • How logs travel from a device to a SIEM, and why delivery delays and dropped events matter.
  • Kerberos basics: tickets, the KDC (Key Distribution Centre) and why timestamps are in the messages.
  • Reading a Windows Security log, starting with events 4624, 4625, 4771 and 4616.
  • Practice: on a lab machine, run the w32tm or timedatectl commands above, then change nothing and just read the output. Note the source and the stratum.

Key takeaways

  1. Clocks drift, and NTP over UDP port 123 keeps them in step with a trusted source.
  2. Stratum shows how far a device is from the reference clock, and 16 means unsynchronised.
  3. Kerberos tolerates 5 minutes of difference by default, and event 4771 with code 0x25 signals clock skew.
  4. Event 4616 records a system time change, and any change not made by LOCAL SERVICE deserves a question.
  5. Compare a log's own time with the SIEM's index time to find hosts whose timestamps you cannot trust.

Time is the thread that ties every log source together. Before you trust a timeline, check the clocks behind it, and make it a habit to write down which time zone every timestamp uses.

Written by Ben, Cybersecurity Student

Keep going

More in SIEM and Tools