6.2. Advanced Configuration

This section discusses advanced configuration options that are probably not needed for the average user. Maybe if you want to provide time service to a larger community, you should consider some of these topics.

6.2.1. Server Selection

6.2.1.1. What is the rule of thumb for number of servers to synchronize to?
6.2.1.2. Should the servers be a mix of primary and secondary servers?
6.2.1.3. How should I provide NTP services for a huge network?

6.2.1.1. What is the rule of thumb for number of servers to synchronize to?

It is entirely up to you and your tolerance for outages. Obviously you have some tolerance, or you would be buying GPS receivers and installing your own stratum-1 servers. But three is a good place to start, and you can progress to three-groups-of-three if you feel the need. Remember that network outages are at least as likely as timeserver outages, so if you only have one network path to the outside world then adding a lot more timeservers doesn't really improve your reliability (your ISP is the single-point-of-failure).

6.2.1.2. Should the servers be a mix of primary and secondary servers?

Probably not. The secondaries are good enough for almost everybody. If you care about the small differences in accuracy/precision between the primaries and the secondaries (and you must be close enough, topology-wise, to even see the difference) then you should buy some GPS receivers.

6.2.1.3. How should I provide NTP services for a huge network?

For a huge network you should provide enough redundancy while avoiding a single point of failure. The following discussion will be based on Figure 5, a configuration that is frequently recommended. I'm not saying it's the only possible configuration, but let's just have a closer look.

Figure 5. Configuration for a huge Network

 1a  1b     1c  1d     1e  1f      outside
. \ / ...... \ / ...... \ / ..............
   2a ---p--- 2b ---p--- 2c        inside
  /|\        /|\        /|\
 / | \      / | \      / | \
3a 3b 3c   3e 3f 3g   3h 3i 3j

Key: 1 = stratum-1, 2 = stratum-2, 3 = stratum-3, p = peer

The example configuration uses six stratum-1 servers (1a ... 1f) to synchronize three stratum-2 servers (2a ... 2c). All servers at stratum two are peers to each other. Each of these stratum-2 servers serve three stratum-3 servers. Clients will be using the servers at stratum three.

Having more than one reference server configured increases reliability and stability of the client (See Q: 5.3.2.). That is why there are two servers for each of the stratum-2 servers. Distributing time horizontally (peering) reduces the amount of traffic to the stratum-1 servers while giving additional redundancy for the stratum-2 servers. The extra layer of stratum-2 servers helps to distribute the load created by lower levels (stratum-3).

If you have a reference clock, you would probably arrange peering with one or more stratum-1 server. For most networks you can probably leave out the third layer (stratum-3) completely.

There's an additional comment by David Dalton:

But my advice is this: if your stratum-N peers all use the same ISP to get to the outside world, then peers are mostly pointless. Your single-point-of-failure is the network path, not the stratum-1 machines themselves. Building huge redundancy into your hierarchy can get very expensive very quickly. Think hard about how much redundancy you really need.

And another comment from Mark Martinec:

I don't find the Figure 5 a good idea. It has a big problem in that stratum-3 servers in the picture all have a single point of failure in their single reference stratum-2 NTP server, not to mention it throws away all the fancy NTP algorithms.

As a fix (and to make it cleaner/leaner), one could strike out the stratum-3 layer completely from the picture and say that each of the company clients will use all three peered company stratum-2 servers as their reference.

If one really needs more fanout (doubtful), one can put back the stratum-3 layer, but with each stratum-3 server referenced to each of the company stratum-2 servers.

6.2.2. Authentication

6.2.2.1. Why Authentication?
6.2.2.2. How is Authentication applied?
6.2.2.3. How do I create a key?
6.2.2.4. How does Authentication work?
6.2.2.5. Can I add Authentication without restarting ntpd?
6.2.2.6. How do I use Public-Key Authentication (autokey)?

6.2.2.1. Why Authentication?

Most users of NTP do not need authentication as the protocol contains several filters against bad time. However, there is still authentication, and its use seems to become more common. Some reasons might be:

  • You only want to use time from trusted sources

  • An attacker may broadcast wrong time stamps

  • An attacker disguise as another time server

6.2.2.2. How is Authentication applied?

NTP uses keys to implement authentication. These keys are used when exchanging data between two machines. As shown in Q: 6.1.3.3. and Q: 6.1.3.4., one of the uses has to do with remote administration. When configuring a server or peer, an authentication key can be specified.

In general, both parties need to know the keys. The keys residing in /etc/ntp.keys typically are unencrypted and thus should be hidden from public.[1] This means, the keys have to be distributed to all communication partners in a secure way. The following example is derived from html/notes.htm:

peer 128.100.49.105 key 22
peer 128.8.10.1     key 4
peer 192.35.82.50   key 6

# path for key file
keys /usr/local/etc/ntp.keys

trustedkey 4 6 14 15 22 # define trusted keys
requestkey 15    # key (7) for accessing server variables
controlkey 15    # key (6) for accessing server variables

authdelay 0.000094      # authentication delay (Sun4c/50 IPX)

The keyword key specifies the key to be used when talking to the specified server. You must trust the key to synchronize time. As authentication involves extra computing, the keyword authdelay specifies this amount of time. In newer versions this calculation is done automatically, while older distributions have a utility named authspeed to determine this number for DES or MD5.

6.2.2.3. How do I create a key?

  • An A key is just a sequence of up to eight ASCII characters (some characters with special meaning can't be used).

  • An M key is a sequence of up to 31 ASCII characters.

  • An S key is a 64 bit value with the low order bit of each byte being odd parity.

  • An A key is a 64 bit value with the high order bit of each byte being odd parity.

  • Now that you know the basics for keys, use a key as good as a password. For an example with some valid keys see Q: 6.1.3.3.. More information can be found in html/confopt.htm and html/notes.htm.

    6.2.2.4. How does Authentication work?

    Basically authentication is a digital signature, and no data encryption (if there is any difference at all). The usual data packet plus the key is used to build a non-reversible magic number that is appended to the packet. The receiver (having the same key) does the same computation and compares the result. If the results match, authentication suceeded.

    6.2.2.5. Can I add Authentication without restarting ntpd?

    Yes and No: You can dynamically add servers that use authentication keys, and you can trust or un-trust any key using xntpdc. You can also re-read the keyfile using the readkeys command. Unfortunately you need basic authentication before using any of these commands (see Q: 6.1.3.3.).

    6.2.2.6. How do I use Public-Key Authentication (autokey)?

    The following material is mostly from an article by Jürgen Georgi, based on release 4.0.99k (autokey v1, rsaref20).[2]

    1. You need extra RSA libraries (like rsaref. See README.rsa for instructions.

    2. You need the utility ntp_genkeys to generate key-pairs and the Diffie-Hellman parameters file. All generated files have a timestamp suffix, it is recommended to install a symlink from the default name (without the timestamp extension) to the actual file:

      $ ntp_genkeys
      Generating MD5 key file...
      Generating RSA public/private key pair (512 bits)...
      Generating Diffie-Hellman parameters (512 bits)...
      $ ls
      ntp.keys.3174020162               ntpkey_dh.3174020162
      ntpkey.3174020162                 ntpkey_nops.BelWue.DE.3174020162
      $ ln -s ntp.keys.3174020162 ntp.keys
      $ ln -s ntpkey.3174020162 ntpkey
      $ ln -s ntpkey_dh.3174020162 ntpkey_dh
      $ ln -s ntpkey_nops.BelWue.DE.3174020162 ntpkey_nops.BelWue.DE

      In this example, nops.BelWue.DE is the canonical name of the local host. It is automatically appended to the file names by ntp_genkeys. File ntpkey_nops.BelWue.DE contains the public RSA key of host nops.BelWue.DE. File ntpkey contains the private RSA key. Needless to say that ntpkey and ntp.keys must not be world readable.

    3. Create a configuration file ntp.conf and a directory structure like this:

      crypto
      keysdir /etc/ntp/
      keys /etc/ntp/ntp.keys
         ...
      server noc1.belwue.de autokey version 4
      server noc2.belwue.de autokey version 4
      server rustime01.rus.uni-stuttgart.de version 3
      peer nepi.BelWue.DE autokey version 4
         ...
      /etc/ntp/
      /etc/ntp/leap-seconds.3169152000
      /etc/ntp/ntp.keys -> ntp.keys.3174020162
      /etc/ntp/ntp.keys.3174020162
      /etc/ntp/ntpkey -> ntpkey.3174020162
      /etc/ntp/ntpkey.3174020162
      /etc/ntp/ntpkey_dh -> ntpkey_dh.3174020906
      /etc/ntp/ntpkey_dh.3174020906
      /etc/ntp/ntpkey_leap -> leap-seconds.3169152000
      /etc/ntp/ntpkey_nepi.BelWue.DE -> ntpkey_nepi.BelWue.DE.3174020497
      /etc/ntp/ntpkey_nepi.BelWue.DE.3174020497
      /etc/ntp/ntpkey_nops.BelWue.DE -> ntpkey_nops.BelWue.DE.3174020162
      /etc/ntp/ntpkey_nops.BelWue.DE.3174020162

      File leap-seconds.3169152000 was downloaded from ftp://time.nist.gov/pub/. File ntpkey_nepi.BelWue.DE is the public RSA key of peer nepi.BelWue.DE. File ntpkey_dh is the same with all authenticated associations, it must be shared among all clients and servers of a security compartment. It does not matter on which host it was generated. You see that the public RSA keys for to the other two authenticated servers are missing. They autokey mechanism is able to download these keys from the servers over the net.

    4. If authentication is working, your output should be similar to this:

      ntpq> pe
           remote           refid      st t when poll reach   delay   offset  jitter
      ==============================================================================
       LOCAL(1)        LOCAL(1)         6 l   58   64  377    0.000    0.000   0.000
      +noc1.belwue.de  .DCFp.           1 u  415 1024  377    2.071    4.886   0.020
      +noc2.belwue.de  .DCFp.           1 u  520 1024  377    1.936    4.891   0.016
      *rustime01.rus.u .DCFp.           1 u  422 1024  377    3.855    3.829   0.037
      -nepi.BelWue.DE  rustime01.rus.u  2 u  259 1024  376    1.839    8.957   0.217
      ntpq> as
      ind assID status  conf reach auth condition  last_event cnt
      ===========================================================
        1 57740  9014   yes   yes  none    reject   reachable  1
        2 57741  f4f4   yes   yes   ok   candidat   reachable 15
        3 57742  f4f4   yes   yes   ok   candidat   reachable 15
        4 57743  9634   yes   yes  none  sys.peer   reachable  3
        5 57744  f334   yes   yes   ok    outlyer   reachable  3
      ntpq> rv
      status=06f4 leap_none, sync_ntp, 15 events, event_peer/strat_chg,
      version="ntpd 4.0.99k-r Thu Jul 27 15:41:30 MET DST 2000 (7)",
      processor="sun4u", system="SunOS5.6", leap=00, stratum=2, precision=-15,
      rootdelay=3.855, rootdispersion=25.972, peer=57743,
      refid=rustime01.rus.uni-stuttgart.de,
      reftime=bd4d0006.7ba24894  Tue, Aug 22 2000 15:35:02.482, poll=10,
      clock=bd4d01be.a8915bdd  Tue, Aug 22 2000 15:42:22.658, state=4,
      phase=4.548, frequency=7.357, jitter=1.913, stability=0.016,
      hostname="nops.BelWue.DE", publickey=3174020162, params=3174020906,
      refresh=3175878685, leaptable=3169152000, tai=32

    Let's add a final quote from Jürgen Georgi: "This setup works with ntp-4.0.99k. I could not get it working with 4.0.99i. If my description contains errors, please let me know."

    See also Q: 6.1.3.6.. For those interested in the details, let's quote Professor David L. Mills: "The latest Autokey draft is at http://www.ietf.org/internet-drafts/draft-ietf-stime-ntpauth-04.txt. Almost all the cryptographic means is in ./ntpd/ntp_crypto.c in the latest distribution. (...)"

    The following procedure for configuring autokey v2 had been extracted from an article posted by Steve Kostecke in news://comp.protocols.time.ntp. You should also have a look at http://support.ntp.org/bin/view/Support/ConfiguringAutokey for a possibly updated procedure. The original procedure has been updated from that URL.

    There are several Identity Schemes available in the NTP Reference Implemenation: Private Certificate (PC), Schnorr (IFF) scheme, Guillou-Quisquater (GQ) scheme, and Mu-Varadharajan (MV) scheme. Identity Scheme selection and enforcement of crypto use (with restrict statements) are beyond the scope of this document. See Identity Schemes (http://www.eecis.udel.edu/~mills/keygen.html). Using just one Identity Scheme should be sufficient.

    1. Set up the Server

      1. Create a directory for the NTP Keys (e.g. /etc/ntp). That directory should be readable only by the user that runs ntpd (e.g. root).

      2. Add the following lines to the configuration file (e.g. /etc/ntp.conf):

        crypto pw serverpassword
        keysdir /etc/ntp
      3. Add autokey at the end of of any broadcast line in /etc/ntp.conf if you want autokey to be used.

      4. Generate Server Parameters

        1. IFF Parameters

          Generate the IFF parameters (server key) using the following commands (-T generates a trusted certificate):

          cd /etc/ntp
          ntp-keygen -T -I -p serverpassword

          That command creates three files (and a link to each), all starting with ntpkey_, containing the hostname of the machine and the time of creation as decimal number (like ntpkey_IFFpar_aax264.3372523673):

          IFFpar

          Encrypted DSA private key (server and client keys)

          RSAkey

          Encrypted RSA private key

          RSA-MD5cert

          X.509 self-signed (trusted) CA certificate of the host (valid for one year)

          IFF Group Keys for each client may be extracted and mailed using the following commands:

          cd /etc/ntp
          ntp-keygen -e -q serverpassword -p clientpassword | \ mail timelord@client.domain

          IFFkey

          Encrypted DSA private key

        2. GQ Parameters

          Generate the GQ parameters with the following commands:

          cd /etc/ntp
          ntp-keygen -T -G -p serverpassword

          GQ parameters are shared between clients.

          GQpar

          Encrypted RSA private key (server and client key)

        3. MV Parameters

          Generate the MV parameters with the following commands (N-1 client keys will be generated):

          cd /etc/ntp
          ntp-keygen -V N -p serverpassword
          ln -s ntpkey_MVpar_server.3301144193 ntpkey_mv_server
          ln -s ntpkey_MVkey1_server.3301144193 ntpkey_mvkey_server

          MVkeyn

          Encrypted DSA private key

          MVpar

          Encrypted DSA private key

          RSAkey

          Encrypted RSA private key

          RSA-MD5cert

          X.509 self-signed CA certificate of the host (valid for one year)

        4. Multiple sets of parameters may be generated using the following commands:

          cd /etc/ntp
          ntp-keygen -V 5 -q serverpassword
          ntp-keygen -T -I -q serverpassword
          ntp-keygen -T -G -q serverpassword
      5. Restart ntpd on the server. Watch the output of ntpq -p to make sure that the server is able to start.

      6. The server certificate should be updated monthly with the following command (only the certificate is renewed, not the key):

        cd /etc/ntp
        ntp-keygen -T -q serverpassword

        GQpar

        Encrypted RSA private key

        RSA-MD5cert

        X.509 self-signed (trusted) CA certificate of the host (valid for one year)

    2. Set up the Client

      1. Create a directory for the NTP Keys (e.g. /etc/ntp). That directory should only be readable by the user running ntpd (e.g. root).

      2. Add the following lines to the configuration file (e.g. /etc/ntp.conf):

        crypto pw clientpassword
        keysdir /etc/ntp

        Append autokey to the server line for the time-server that you want to authenticate with Autokey:

        server host.I_have_the_key_for autokey
      3. Generate the client key/certificate with the following commands:

        cd /etc/ntp
        ntp-keygen -H -p clientpassword

        RSAkey

        Encrypted RSA private key

        RSA-MD5cert

        X.509 self-signed CA certificate of the host (valid for one year)

      4. Install Group/Client Keys

        1. IFF Group Keys

          Obtain the IFF group key (see step 1) via a Secure Web Form[3] or some other secure means, copy the key file to the keysdir, and create the standard symbolic link:

          cd /etc/ntp
          ln -s ntpkey_IFFpar_server.3301264563 ntpkey_iff_server

          As the parameter file contains both, server and client keys, one might prefer to create only the client key on the server using ntp-keygen -e -q serverpassword (according to keygen.html).

        2. GQ Group Keys

          Obtain the GQ group key from step 2, copy the key file to the keysdir, and create the standard symbolic link:

          cd /etc/ntp
          ln -s ntpkey_GQpar_server.3301145293 ntpkey_gq_server
        3. Obtain the MV client key (see step 3), copy the key file to the keysdir, and create the standard symbolic link:

          cd /etc/ntp
          ln -s ntpkey_MVkey1_server.3301144193 ntpkey_mvkey_client
      5. Restart ntpd on the client. Watch the output of ntpq -p to make sure that the client is able to start and sync with the server.

      6. The client certificate has a limited validity and should be updated periodically using the following command (only the certificate is renewed, not the key):

        cd /etc/ntp
        ntp-keygen -q clientpassword

        RSA-MD5cert

        X.509 self-signed CA certificate of the host (valid for one year)

    The procedure above has been verified successfully using ntpd versions 4.2.0a@1.1320-o and 4.2.0a@1.1190-r running on Linux.

    Warning

    As the hostname is used when generating keys, keys must be created on the right machine, and the the network name must match the host name for authentication to succeed in ntpd. Options -i name and -s name of ntp-keygen allow to specify a different subject and issuer respectively however.

    6.2.3. Broadcasting, Multicasting, and Manycasting

    With Broadcasting and Multicasting several clients can be addressed with a single packet transmitted by the server.

    1. Broadcasting
    6.2.3.1.1. How do I configure a Broadcast Server?
    6.2.3.1.2. How do I configure a Broadcast Client?
    6.2.3.1.3. Why doesn't Broadcasting work with LCL?
    2. Multicasting
    6.2.3.2.1. How do I configure Multicast Servers and Clients?
    3. Manycasting
    6.2.3.3.1. What is Manycasting at all?
    1. Broadcasting

    6.2.3.1.1. How do I configure a Broadcast Server?

    A line like broadcast 128.4.2.255 enables periodic sending of broadcast packets containing the current time as long as the server's clock is synchronized. The period may be influenced by the minpoll option. Packet forwarding can be limited by specifying the ttl option. Make sure you are using the correct broadcast address for your subnet.

    For the client use a line containing broadcastclient.

    6.2.3.1.2. How do I configure a Broadcast Client?

    Using the line broadcastclient will enable listening to broadcasts. As anybody can send out any broadcasts, use of authentication is strongly advised. In version 3 you would have to define the broadcastdelay to compensate the network delay. In version 4 the client actively will query a broadcasting server to calibrate the delay. More details can be found in file html/assoc.htm (Association Management).

    6.2.3.1.3. Why doesn't Broadcasting work with LCL?

    Before continuing, make sure you read and understood Q: 7.1.1. and Q: 6.1.2.1.. As you shouldn't broadcast bad time, a prefer keyword is required when using LCL.

    2. Multicasting

    6.2.3.2.1. How do I configure Multicast Servers and Clients?

    Multicasting is configured just like broadcasting, but instead of using a broadcast address, a multicast group address (class D) is used.

    3. Manycasting

    6.2.3.3.1. What is Manycasting at all?

    This is an explanation by Professor David L. Mills: "Manycast only works in multicast mode. It uses an expanding-ring search by adjusting the TTL field. This doesn't make sense in broadcast mode, since broadcast packets do not span subnets. It might in fact be useful to implement manycast in broadcast mode without the search, but that is rather far down the to-do list." (...) "Only the * and + tattletales indicate a candidate survivor. Note that one of your servers is in process of going away, another coming onboard. This is a normal situation when first coming up and when the signatures are refreshed once per day. I assume you are using autokey; if not, no promises at all."

    So basically it's a mechanism to automatically configure servers on a nearby network. Compared to broadcasting and multicasting, manycasting uses the normal server keyword, but with a multicast group address (class D) on the client. Manycast servers use the keyword manycastserver. As for broadcasts and multicasts, manycast associations on the client may come and go over time.

    6.2.4. PPS Synchronization

    PPS synchronization is an option that is neither necessary nor available for all operating systems. Still it brings many benefits if used, so it's discussed here.

    As stated before, network connections suffer from random delays. Even for local reference clocks the exact point in time to which some time message belongs is difficult to determine. Delays imposed by the operating system is another issue to deal with.

    Therefore some means to improve the situation were thought of:

    Basically this is what all the PPS discussion is about. So let's add some questions.

    1. Before you start
    6.2.4.1.1. What are the components required to use PPS synchronization?
    2. Verification
    6.2.4.2.1. So I think I have all required components ready, how will I see that everything is working?
    3. Special Drivers
    6.2.4.3.1. What is that ATOM or PPS peer?
    6.2.4.3.2. How do I use PPS with the Motorola Oncore driver?
    6.2.4.3.3. How do I use PPS with NMEA driver
    4. Configuration
    6.2.4.4.1. What changes are required in ntp.conf?
    5. Software Interfaces
    6.2.4.5.1. What is that PPS API?
    6. Hardware Interfaces
    6.2.4.6.1. What is the shortest Width for a Pulse connected to the DCD Pin of an RS-232 Interface?
    1. Before you start

    6.2.4.1.1. What are the components required to use PPS synchronization?

    The following items are needed in oder to use PPS synchronization:

    • You need a high precision signal that can be connected to the computer running NTP. Usual sources of such PPS signals are quality reference clocks that feature such an output.

    • Your operating system must support processing of PPS signals. Most operating systems that come with source code (such as Linux and FreeBSD) can be modified to support PPS processing. As a matter of fact someone else probably did it already so you just need to install and configure the software.

      Most operating systems supporting PPS do not only have some programming interface to read timestamps, but they also implement the NTP kernel clock model with special PPS processing options. See also Q: 5.2.4.3..

    • The NTP software must also be configured to recognize and use PPS processing. Usually the software's autoconfigure process will detect the presence of PPS processing capabilities.

    • Finally you should prepare your configuration file ntp.conf to work with PPS.

    2. Verification

    6.2.4.2.1. So I think I have all required components ready, how will I see that everything is working?

    The pleasant part of this answer is that there are tools included in the standard NTP software that makes this an easy task. The less pleasant part is that there is no single way to enable PPS detection for each operating system. However the new PPS API (see Q: 6.2.4.5.1.) may change things in a positive way.

    Let's start with the easier part using standard tools:

    1. There are two means to look at the NTP kernel clock: ntptime and ntpdc -c kerninfo. As ntptime is a little more verbose, let's consider it.

      windl@elf:~ > ntptime
      ntp_gettime() returns code 0 (OK)
        time bd6b9cf2.9c3c6c60  Thu, Sep 14 2000 20:52:34.610, (.610297702),
        maximum error 3480 us, estimated error 0 us.
      ntp_adjtime() returns code 0 (OK)
        modes 0x0 (),
        offset 1.658 us, frequency 17.346 ppm, interval 128 s,
        maximum error 3480 us, estimated error 0 us,
        status 0x2107 (PLL,PPSFREQ,PPSTIME,PPSSIGNAL,NANO),
        time constant 6, precision 3.530 us, tolerance 496 ppm,
        pps frequency 17.346 ppm, stability 0.016 ppm, jitter 1.378 us,
        intervals 57, jitter exceeded 29, stability exceeded 0, errors 0.

      The above command has been run on Linux version 2.2.16 with PPSkit-1.0.0. That combination features PPS processing and a kernel clock using nanoseconds.

      The first thing you should look at is the status (0x2107 in our case). The magic words in parentheses explain the meaning of the individual bits. The important bit for now is PPSSIGNAL. That bit is set directly by the operating system and says a PPS signal has been detected.

    2. Now that pulses are detected, let's see whether they are good ones. For that purpose we read some additional numbers about the kernel clock's calibration process:

      • intervals 57 says that there were 57 calibration intervals. When PPS pulses are arriving, this number should increase. Each frequency adjustment requires a good calibration interval. The length of the current calibration interval can be found as interval 128 s (128 seconds is the default maximum length). Remaining numbers count abnormal conditions as explained below.

      • jitter exceeded 29 means that there were 29 pulses that arrived at a time when they were not considered good (However, completely bad pulses are not counted here). This can mean that the pulses were out of range, or that the system clock was read badly when the pulses arrived. The algorithm dynamically adjusts the threshold for jitter. Occasional jitter is allowed and should not worry you. If significant jitter is detected, the flag PPSJITTER is set in addition.

      • stability exceeded 0 is the number of calibration intervals that would result in a correction larger than considered valid (The default limit quite high).

        Validity is a bit fuzzy here, but it means that the frequency change would be more than the tolerance. You can find the tolerance coded into your operating system from tolerance 496 ppm, but it's not guaranteed that the value is specific to your hardware.

        Stability should not be exceeded during normal operating conditions. Upon detection of that error the flag PPSWANDER is set.

      • Finally errors 0 indicates the number of calibration intervals where pulses were missing or completely out of bounds. In these cases the flag PPSERROR is set. During normal operation that number should not increase.

      Those numbers are only reset when the machine is booted.

    3. If you did not find an error so far, your PPS configuration should work! You can inspect some additional performance indicators:

      • stability 0.016 ppm is an averaged value for the last frequency corrections made (actually it's an instability). Basically a small value indicates that both, your operating system's clock, and your external PPS signal are stable. As mentioned before, temperature changes affect the average PCs a lot (See also Q: 5.1.3.1.). The sample above was taken after running the system for about one hour; you should expect a value below 0.1 ppm for a stable system.

      • jitter 1.378 us is also an averaged value. It indicates how much the individual pulses vary from second to second (as measured by the operating system's clock). This value will vary due to system load and interrupt latency. A few microseconds are probably fine, but a few milliseconds definitely are not!

    This completes the basic checks for PPS configuration. In the case above the NTP daemon is also working and using the data provided from the operating system kernel. I'll complete the description of the remaining output:

    • pps frequency 17.346 ppm indicates the current correction value for the clock frequency derived from the PPS signal. Positive values indicate that your clock is too slow compared to the PPS. If flag PPSFREQ is set, that frequency correction is used for correcting the kernel clock. The NTP daemon will set this flag if the PPS parameters seem valid.

    • offset 1.658 us shows the last measured offset correction for the system clock. If flag PPSTIME is set, that offset is derived from the offset of the PPS pulse every second, and otherwise it's updated through ntp_adjtime() from the application. A positive value for offsetmeans that the system clock is behind the reference time.

      PPS pulses add further corrections while the kernel clock tries to consume this offset by correcting the time accordingly. Formerly the offset was updated every 16 seconds by the kernel, but recently it's updated every second.

    3. Special Drivers

    6.2.4.3.1. What is that ATOM or PPS peer?

    Even when the kernel clock uses PPS signals to calibrate, the NTP daemon will still use the usual offsets of some reference clock. As it is desirable to use the offsets of the PPS pulses, there is a pseudo clock driver to do that. That driver needs to know the interface specific to the platform to get the time stamps of the PPS pulses.

    That driver is called ATOM or PPS, and it can be configured just as any other reference clock. The difference is that PPS can only be used in combination with another preferred time reference. As soon as the preferred time reference is used for synchronization, the ATOM driver becomes reachable, and it will eventually be used as primary synchronization source (See also Why should I have more than one clock?). A PPS peer will be handled specially so that other time offsets are not considered. The command ntpq -c peer -c as -c rl will print something like:

         remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    +GENERIC(1)      .GPS.            0 l   48   64  377    0.000    0.025   0.001
    oPPS(1)          .PPS.            0 l   17   64  377    0.000    0.027   0.000
    
    ind assID status  conf reach auth condition  last_event cnt
    ===========================================================
      1 57300  9434   yes   yes  none  candidat   reachable  3
      2 57301  9714   yes   yes  none  pps.peer   reachable  1
    
    status=2194 leap_none, sync_atomic/PPS, 9 events, event_peer/strat_chg,
    version="ntpd 4.0.99k Sun Sep 10 19:22:28 MEST 2000 (5)",
    processor="i586", system="Linux2.2.16-NANO", leap=00, stratum=1,
    precision=-16, rootdelay=0.000, rootdispersion=1.831, peer=57301,
    refid=PPS, reftime=bd6b94f2.272b8844  Thu, Sep 14 2000 20:18:26.153,
    poll=6, clock=bd6b952d.da89dadf  Thu, Sep 14 2000 20:19:25.853, state=4,
    phase=0.005, frequency=16.984, jitter=0.000, stability=0.043

    6.2.4.3.2. How do I use PPS with the Motorola Oncore driver?

    Considering the configuration below, John Hay wrote:

    (...)The Oncore driver directly manages the PPS stuff, so you only need the first line (server 127.127.30.0 prefer) in the config file. The rest is not needed to have a functional Oncore refclock.

    Some options like the flag3 and enable pps stuff is remnants of the xntp3 days I think and isn't used by ntp4 anymore. The "pps ..." line is only useful for drivers like the NMEA driver, that don't manage the PPS signals themselves, but use the global serial port access routines and even in that case the line must precede the server line for the same device in the config file.

    server 127.127.30.0 prefer
    fudge 127.127.30.0 stratum 0
    
    server 127.127.22.1                     # ATOM(PPS)
    fudge 127.127.22.1 flag3 1              # enable PPS module
    pps /dev/oncore.pps.0 assert hardpps    # PPS device

    Note: As documented in PPS Clock Discipline (driver8.htm) of ntp-4.1.1a, flag2 controls the edge of the PPS signal being used. Thus the pps keyword seems obsolete by now.

    Maybe it should also be noted here that a site survey can take significant time to finish. Terje Mathisen says: "My survey (under Linux) took about 36 hours, I also gave up a couple of times before allowing it to run to completion."

    6.2.4.3.3. How do I use PPS with NMEA driver

    In NTP versions after 4.0.99k23 is NMEA driver atomized what means that for PPS processing we don't need neither ATOM driver nor PPS command in ntp.conf.

    Here is sample ntp.conf:

    server 127.127.20.0     # NMEA driver
    fudge  flag3 1          # enable kernel PPS discipline
    4. Configuration

    6.2.4.4.1. What changes are required in ntp.conf?

    Unfortunately PPS processing is a little messy (see also What is that PPS API?). Nevertheless I'll give an example that works.

    Example 4. Using a PPS Signal

    This example works for ntp-4.0.99k together with PPSkit-1.0.0 on my Linux PC. I'll only show the significant parts of /etc/ntp.conf[4] .

    pps /dev/refclock-1 assert hardpps	# PPS device (ntpd-4.0.97 and above)
    
    server 127.127.8.1 mode 135 prefer	# Meinberg GPS167 with PPS
    fudge 127.127.8.1 time1 0.0042		# relative to PPS for my hardware
    
    server 127.127.22.1 			# ATOM(PPS)
    fudge 127.127.22.1 flag3 1		# enable PPS API

    When starting, the following things happen:

    1. The clock GENERIC(1) becomes reachable while PPS is used to update the kernel variables described in Q: 6.2.4.2.1..

    2. The configured clock is selected as synchronization source, and status changes to 0x2143 after a while. At that time PPS(1) also becomes reachable. During that time status changes to 0x2107, and offset shows current offsets from PPS.

    3. Eventually PPS(1) becomes PPS peer.

    5. Software Interfaces

    6.2.4.5.1. What is that PPS API?

    As seen above, the programming interface specific to the operating system and platform is a messy thing. Therefore some people decided to make a common programming interface named PPS API. In March 2000 that draft was accepted as an informational RFC (See Table 4 for related RFCs). The functions of the API include:

    • Routines to enable capturing of external events on a specified device (if supported).

    • Routines to query the last captured time stamps and associated event counters.

    • Routines to change operating parameters like compensating processing delays, select polarity of the PPS signal, etc.

    • Routines to control automatic processing of detected events by a kernel consumer in the kernel of the operating system.

    As the API is still quite new, there are only very few implementations ([RFC 2783] says: "Several available implementations of this API are listed at http://www.ntp.org/ppsapi/PPSImpList.html. Note that not all of these implementations correspond to the current version of the specification.").[5]

    Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

    The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

    6. Hardware Interfaces

    6.2.4.6.1. What is the shortest Width for a Pulse connected to the DCD Pin of an RS-232 Interface?

    That depends. The higher the quality your serial port is, the longer pulses will be needed. This is because of the ESD protection of the chip.

    26 usec should be about the bit time for a 38400 serial line, so you could connect it to the RxD line instead and see if you receive characters that way when the port is set for 38400 or faster.

    Another thing to try is to configure the serial port for 115200, some of the chips base their deglitching on the baud rate, often requiring a full symbol before they react.

    6.2.5. Automatic Configuration

    If many systems have to be configured in a similar way, there is a desire to automate the process. We try to give some useful hints here.

    1. BOOTP
    6.2.5.1.1. How can I define the address of an NTP server in a BOOTP reply?
    6.2.5.1.2. How do I use information about NTP servers given in a BOOTP reply?
    1. BOOTP

    6.2.5.1.1. How can I define the address of an NTP server in a BOOTP reply?

    The BOOTP protocol is defined in [RFC 1048] (obsoleted by [RFC 2132]). Marc Brett contributed:

    Time ([RFC 868]) servers may be specified in the Vendor Extensions field, Code 4.

    Network Time Protocol (NTP) ([RFC 1305]) servers may be specified in the Application and Service Parameters, Code 42.

    In some popular bootpd, time servers are specified with ts=, but NTP servers are specified with nt=. The latter allows a list of Internet addresses to be specified.

    6.2.5.1.2. How do I use information about NTP servers given in a BOOTP reply?

    If you are using Microsoft Windows or Macintosh, you have to check whether your DHCP client software or your NTP software contains such a feature already. If not, you are probably out of luck.

    If you are using a UNIX system, you may be able to install the needed code yourself. Of course, exactly what is needed depends on your particular software, but if you are comfortable editing the initialization scripts, it shouldn't be too hard.

    In general, UNIX systems implement BOOTP/DHCP via a client daemon which handles the interaction with the DHCP server. But often, the daemon does not itself set up the network interface's address or do any other work based on the DHCP reply. Instead, it stores the information in some suitable place, and activates an initialization script to use that information to adjust the computer's configuration.

    Your first job is to identify the correct script. Start by tracking down the script that brings up the interface during normal startup. That script will have code which activates the DHCP client, and thus must also arrange to activate the script that implements the configuration provided by DHCP.

    For instance, in RedHat Linux 8.0, a network interface is brought up by /etc/sysconfig/network-scripts/ifup, which calls the DHCP client (/sbin/dhclient) at line 188. When dhclient receives a response from the DHCP server, it executes a program specified by the "-sf" argument, or by default, /sbin/dhclient-script. It is that script which adjusts the computer's configuration. In this case, the script is already set up (line 152) to change ntp.conf if the DHCP server provides NTP server information.

    In RedHat Linux 7.x, a network interface is also brought up by /etc/sysconfig/network-scripts/ifup, which calls the DHCP client (/sbin/pump or /sbin/dhcpcd) near line 127. We see from the dhcpcd(8) manual page that dhcpcd revises resolv.conf itself (unless "-R" is specified), but that other DHCP configuration information can be implemented by the script /etc/dhcpc/dhcpcd-interface.exe.

    6.2.6. Offering Time Service

    This section presents some guidelines for offering time service to others.

    6.2.6.1. My NTP Server has a number of IP Addresses for different Nets. Is there any way to request ntpd to attach to a specific Interface?
    6.2.6.2. Should Access be restricted?
    6.2.6.3. What should be done before announcing public NTP service?

    6.2.6.1. My NTP Server has a number of IP Addresses for different Nets. Is there any way to request ntpd to attach to a specific Interface?

    As far as I know, ntpd attaches to all interfaces. What happens if you have virtual adresses (interface aliases) depends on the operating system. For some operating systems ntpd listens to all adresses.

    It is known that the issue is handled sub-optimally, and it's being worked on...

    6.2.6.2. Should Access be restricted?

    First, well, if you don't want to have clients, don't offer the service. But as you want to offer NTP service to others, you should not be afraid of clients. Let me quote from an article in news://comp.protocols.time.ntp written by David Dalton about the subject whether queries with ntpq and ntpdc should be allowed:

    I am somewhat new to the security concerns of public timeservers. Only in the past few weeks did I upgrade my public timeserver at 192.6.38.127 to stratum-1 with a Trimble Palisade GPS receiver. It doesn't have a lot of security right now, but the "xntpdc" reconfiguration functions are restricted. I'm soliciting advice about how to protect myself, although I don't depend on that public timeserver in any way.

    I agree with you that there is no substitute for long-term data for evaluating the stability of a timeserver (and the network between yourself and the timeserver).

    But the query tools allow one to make evaluations without spending a lot of time, because the timeservers themselves have already collected the long-term data. I always want to run "ntpq -p" or "xntpdc -p" on a remote timeserver before committing to it. Very handy.

    Even long-term statistics (gathered by your own client) won't tell you anything about how well the remote server is configured. How many reference clocks does it have? Which reference clocks? How many stratum-1 servers does it have in case the clock(s) fail(s)? Which one of these candidates would you prefer?

    EXAMPLE ONE (BAD)
    -----------------
    [444]ntpq -p fubar.net
         remote      refid      st t when poll reach   delay   offset    disp
    =========================================================================
    *WWVB_SPEC       .WWVB.      0 l   18   16  377     0.00    0.301    1.69
     LOCAL(1)        LOCAL(1)    0 l    1   16  377     0.00    0.000   10.01
    
    EXAMPLE TWO (NOT BAD)
    ---------------------
    [170]ntpq  -p fubar2.net
         remote      refid      st t when poll reach   delay   offset    disp
    =========================================================================
    *WWVB_SPEC(      .WWVB.      0 l   30   16  377     0.00    0.140    2.01
     LOCAL(1)        LOCAL(1)   10 l   13   16  377     0.00    0.000   10.01
    +hpxxxxxxxx      .GPS.       1 u   11   16  376     0.99   -0.708    0.35
    +hpxxxxxxxx      .GPS.       1 u   49   64  377     4.97   -2.680    0.81
     hpxxxxxxxx      xxxxxxxx    3 u  206 1024  377     4.70   -3.010    9.69
     hpxxxxxxxx      xxxxxxxx    3 u   29 1024  377     2.88   -4.287    0.17
    
    EXAMPLE THREE (OUTSTANDING)
    ---------------------------
    [169]ntpq  -p ntp2.usno.navy.mil
         remote      refid      st t when poll reach   delay   offset    disp
    =========================================================================
    +GPS_VME(0)     .USNO.       0 l   15   16  377     0.00   -0.007    0.02
    *GPS_VME(1)     .USNO.       0 l   14   16  377     0.00    0.003    0.02
    +GPS_VME(2)     .USNO.       0 l   13   16  377     0.00    0.028    0.02
    +tick.usno      .USNO.       1 u   45   64  376     1.65    0.032    0.64
    -tock.usno      .USNO.       1 u   16   64  377     1.48   -0.072    0.47
    x204.34.19      .USNO.       1 u   24   64  377   226.88    3.924    1.77
    x204.34.19      .USNO.       1 u 1014 1024  376   249.76   10.737   26.49
     navobs1.g      0.0.0.0     16 -    - 1024    0     0.00    0.000 16000.0

    That article continues:

    You and I might agree that a handheld consumer-grade GPS receiver putting out NMEA data to a small workstation (unix or WinNT) with 500 milliseconds dispersion is a poor excuse for a stratum-1 public timeserver, but that doesn't stop somebody from offering such a server for public use. No sanity checking or evaluation is done on the machines listed at UDelaware (AFAIK).

    Here is what my handheld GPS receiver (400 dollars) looked like:

    [179]ntpq  -p gpstime.net
         remote      refid      st t when poll reach   delay   offset    disp
    =========================================================================
    *GPS_NMEA        .GPS.       0 l    1   64  377     0.00  -226.243  420.08

    Here is a stratum-2 timeserver that has a good pedigree (HP-UX) and is synching to a stratum-1 timeserver that has an HP GPS clock, but the stratum-2 machine is having problems because the network is _very_ congested (and it has no backup sources):

     remote            refid      st t when poll reach  delay   offset disp
    ==========================================================================
     big_srv         17.8.5.7      2 u    3  512   17   312.87 -249.15 1960.85

    Here are some results from well configured public timeservers that I have surveyed at various times. It is interesting to work your way through the stratum-1 list at UDelaware and see a lot of timeservers this way.

    [165]ntpq  -p ntp-cup.external.hp.com
         remote      refid      st t when poll reach   delay   offset    disp
    =========================================================================
    *REFCLK(29,1)    .GPS.       0 l   21   32  377     0.00    0.014    0.02
    +bigben.cac.wash .USNO.      1 u  115  128  377    38.48   -0.292    0.46
    +clepsydra.dec.c .GPS.       1 u    2  128  377     6.94    0.044    0.21
    -clock.isc.org   .GOES.      1 u  381 1024  377     6.29   -3.159    0.11
     hpsdlo.sdd.hp.c bigben.ash  2 u   25   32  125    53.68   -9.817    3.69
    -tick.ucla.edu   .USNO.      1 u   70  128  377    19.18   -0.894    0.38
    -usno.pa-x.dec.c .USNO.      1 u   39  128  377     7.05   -0.434    0.26

    So there are valid arguments for allowing some standard queries from prospective or active NTP clients. On the other hand there are also arguments for restricting access:

    • Configuration changes (see also How do I use authentication keys?) should be restricted to machines within the own administrative domain at least.

    • You might consider the possibility that a security hole is found in some software, and that that hole could be exploited to do bad things to your server. Therefore you could restrict or enable certain ranges of IP addresses.

    6.2.6.3. What should be done before announcing public NTP service?

    As with any service offered in the Internet, there is a potential to do something stupid. You are strongly advised to do some monitoring of your server before going public (See Section 8.1).

    Once you are satisfied with the performance data, you should also consider the following questions:

    • Does my server have an offset and stability better or equal to other servers at the same stratum?

    • Does my server have redundant or highly available time sources (reference clocks or peers)?

    • Did you arrange peering with at least one other server at the same or at an even better stratum?

    • Do I want to serve possibly hundreds of clients, most of them unknown to me?

    • Can my connection to the Internet satisfy the demands for NTP service (good network response times and very few dropped packets)?

    • Is the server machine highly available (Does it start up automatically after a failure), and is there a contact person in case of problems?

    • Are there plans to continue the service for at least six months?

    If you answered any of the preceeding questions with "No", you should re-consider your decision of offering public time service.

    Notes

    [1]

    There is a recent, but not yet completed project to provide a public keys mechanism for NTP.

    [2]

    autokey v2 is expected to use OpenSSL and thus a different protocol instead.

    [3]

    Web Form for client IFF keys (https://www.kostecke.net/crypto.php)

    [4]

    See note for Q: 6.2.4.3.2..

    [5]

    Jeffrey Mogul suggested the following procedure to access the final draft for the PPS API:

    The easiest path to finding drafts is http://www.ietf.org

    -> Internet-Drafts
    -> Internet Drafts Index
    -> Individual Submissions (or whatever working group)

    and then wait for a huge file to download, so that you can search it for "PPS", which currently leads to: http://www.ietf.org/internet-drafts/draft-mogul-pps-api-05.txt

    [RFC 2783] is actually based on draft-mogul-pps-api-06.txt and should be available as ftp://ftp.isi.edu/in-notes/rfc2783.txt.