config agent: reduce full config check frequency from 5s to 1m and compare hashes instead

[nikw9944]

Summary

• Instead of applying config to the device every 5s, apply it only when the config hash changes, or when it has been 60s since it was last applied.

Changes

Controller

• Refactor GetConfig by extracting config generation into reusable helper functions
• Add architecture documentation with sequence diagram

Agent

• Refactor main loop
• Implement a simple caching scheme for device config
• Instead of apply the full config from the controller every 5 seconds, only apply it if it has changed
• Also apply the full config after cache timeout (default 60s)

Testing Verification

• Added unit tests for controlplane/agent/cmd/main.go
• No functionality has changed so e2e tests should run as-is

[nikw9944]

Here's a before and after comparison of total CPU usage by Arista EOS ConfigAgent when running all tests in parallel.

Before change:

     1400 +---------------------------------------------------------------------------------------------------------+
          |                 +               A+                 +                 +                +                 |
          |                                 *                                                                       |
     1200 |-+                              **                                                                     +-|
          |                                * *                                                                      |
          |                               *  *                                                                      |
          |                               A  *                                                                      |
     1000 |-+                             *  *                                                                    +-|
          |                               *   *                                                                     |
          |                               *   *                                                                     |
      800 |-+                            *    A        A                                                          +-|
          |                              *     A       *                                                            |
CPU %     |                              *      *A    * *                                                           |
      600 |-+                            *        *   * A                                                         +-|
          |                              *        *   *  *                                                          |
          |                              *         A *    A  *A                *A*AA*A*A*A*AA*A*A*A*AA*A*A*A*A      |
      400 |-+                           *           *A     *A  *AA*A*A*A*AA*A*A                                   +-|
          |                             *                                                                           |
          |                             A                                                                           |
          |                            *                                                                            |
      200 |-+                         A                                                                           +-|
          |                           *                                                                             |
          |                 +        *       +                 +                 +                +                 |
        0 +---------------------------------------------------------------------------------------------------------+
          0                100              200               300               400              500               600
                                                       Elapsed (seconds)

After change:

     1400 +---------------------------------------------------------------------------------------------------------+
          |                 +                +     A           +                 +                +                 |
          |                                        **                                                               |
     1200 |-+                           A         * *                                                             +-|
          |                             *         *  *                                                              |
          |                            * *        *  A                                                              |
          |                            * *        *   *                                                             |
     1000 |-+                          *  *      *    *                                                           +-|
          |                           *   A      A     *                                                            |
          |                           A   *      *     A                                                            |
      800 |-+                         *    *    *       A                                                         +-|
          |                           *    *    *        *A                                                         |
CPU %     |                           *     **A *          *                                                        |
      600 |-+                         *     A * *           A                                                     +-|
          |                          *         *             *A                                                     |
          |                          *         A               *A *A                                                |
      400 |-+                        *                           A  *A*A                                          +-|
          |                          *                                  *AA*A*A                                     |
          |                        A*A                                         *A*AA*A*A*A                          |
          |                       *                                                       *A*AA*A*A*A*AA*A*A        |
      200 |-+                    A                                                                                +-|
          |                     *                                                                                   |
          |                 +   *            +                 +                 +                +                 |
        0 +---------------------------------------------------------------------------------------------------------+
          0                100              200               300               400              500               600
                                                       Elapsed (seconds)

Note that the peak usage is about the same, but the steady state usage (after all tests have run and the config agents are just polling) is reduced from about 400% (consuming 4 cores) to about 200% (consuming 2 cores) across 29 EOS containers.