Log Collection

Software processes generate logs which provide insight into the internal workings of the application. As a software system becomes more complex, distributed across multiple devices, there would be hundreds of processes generating millions of log entries. For example, laser sensor device drivers may generate logs including raw point cloud data. At the same time, autopilot software for autonomous drones may dump pathfinding graphs into the logs.

In order to diagnose and debug faults in software, navigating and understanding the vast amounts of log data is necessary in order to find the root cause of failure.

PSYGIG Triage SDK simplies the log collection process by providing a unified logging facility that consolidates logs from multiple sources. The logs can then be pushed to any supported output plugin. The Log Collection API includes many features suitable for integration into embedded systems:

  • Works on existing applications; no need to recompile source code
  • Seamless integration with third party logging platforms and databases
  • Automatically monitor and collect log files generated by known applications
  • Use the built-in logging facility for your application's own logging needs
  • Export logs to a variety of file formats (eg. JSON, CSV, HTML)

Known Log Sources

The Log Collection API provides facilities to discover and monitor log files from the following known applications:

注釈

Support for application log files in this list is a work-in-progress.

This list is constantly expanding. Please if there are log files from an application that you would like to be supported.

unix syslogd (eg. auth, daemon, kern, syslog)
unix journald
unix rsyslogd
unix syslog-ng
apache Apache HTTP Server
tomcat Apache Tomcat
zookeeper Apache Zookeeper
kafka Apache Kafka
nginx Nginx
mysql MySQL
postgresql PostgreSQL
mongodb MongoDB
couchdb CouchDB
redis Redis
aws Amazon AWS
docker Docker
django Django
jenkins Jenkins
haproxy HAProxy
nagios Nagios
cassandra Cassandra
sinatra Sinatra
varnish Varnish

Using psymon

Without modifying your existing source code, you can instantly enable collection of known application logs to your application by using the psymon script to run your program. For example:

psymon --log -c "/path/to/your/app --arg1 --arg2 --arg3"

See psymon man page for a complete list of options.

注釈

psymon is a work-in-progress. Please if you are interested in beta-testing this feature.

Using PSYGIG SDK API

The Log Collection API can be accessed by including a single header file and linking the libpsygig shared library. See Advanced Setup - Building your application with libpsygig for details.

Real-time monitoring of known log files

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#include <psygigsdk.h>

int main(int argc, char **argv)
{
    psygig_agent_handle     pah;  // handle to psygig agent

    /* ... Initialize code omitted for simplicity ... */

    // Search for known log files
    char**      logfiles = NULL;      // Array of known log files
    uint32_t*   logfileflags = NULL;  // Array of log flags
    int         numfiles = 0;         // Number of log files found

    rc = psygig_log_glob_known_log_files(&logfiles, &logfileflags, &numfiles);

    // Monitor all log files found in real-time, at a period of 1 second
    int i;
    for (i = 0; i < numfiles; i++)
    {
        char *logfile = logfiles[i];
        rc = psygig_log_monitor_log_file(pah, logfile, 1000);
    }

    // Free memory allocated to arrays
    psygig_log_glob_free(logfiles, logfileflags, numfiles);

    /* ... Cleanup code omitted for simplicity ... */
}

Sample syslog JSON output

{
    "Log": {
        "Duration": 1,
        "Timestamp": 1544063871,
        "TimeOfDay": "2018-12-06 11:37:51",
        "Text": "Dec  6 11:37:50 ubuntu kernel: [18825.174713] [drm:vmw_cmdbuf_work_func [vmwgfx]] *ERROR* Command \"SVGA_3D_CMD_DX_SET_SHADER_RESOURCES\" causing device error."
    }
},
{
    "Log": {
        "Duration": 1,
        "Timestamp": 1544063871,
        "TimeOfDay": "2018-12-06 11:37:51",
        "Text": "Dec  6 11:37:50 ubuntu kernel: [18825.174721] [drm:vmw_cmdbuf_work_func [vmwgfx]] *ERROR* Command buffer offset is 48"
    }
},
{
    "Log": {
        "Duration": 1,
        "Timestamp": 1544063871,
        "TimeOfDay": "2018-12-06 11:37:51",
        "Text": "Dec  6 11:37:50 ubuntu kernel: [18825.174739] [drm:vmw_cmdbuf_work_func [vmwgfx]] *ERROR* Command size is 20"
    }
}