Log Collection

Software processes generate logs which provide insight into the internal workings of the application. As a software system becomes more complex, distributed across multiple devices, there would be hundreds of processes generating millions of log entries. For example, laser sensor device drivers may generate logs including raw point cloud data. At the same time, autopilot software for autonomous drones may dump pathfinding graphs into the logs.

In order to diagnose and debug faults in software, navigating and understanding the vast amounts of log data is necessary in order to find the root cause of failure.

PSYGIG Mobility IoT SDK simplies the log collection process by providing a unified logging facility that consolidates logs from multiple sources. The logs can then be pushed to any supported output plugin. The Log Collection API includes many features suitable for integration into embedded systems:

  • Works on existing applications; no need to recompile source code

  • Seamless integration with third party logging platforms and databases

  • Automatically monitor and collect log files generated by known applications

  • Use the built-in logging facility for your application’s own logging needs

  • Export logs to a variety of file formats (eg. JSON, CSV, HTML)

Known Log Sources

The Log Collection API provides facilities to discover and monitor log files from the following known applications:

Note

Support for application log files in this list is a work-in-progress.

This list is constantly expanding. Please contact us if there are log files from an application that you would like to be supported.

unix

syslogd (eg. auth, daemon, kern, syslog)

unix

journald

unix

rsyslogd

unix

syslog-ng

apache

Apache HTTP Server

tomcat

Apache Tomcat

zookeeper

Apache Zookeeper

kafka

Apache Kafka

nginx

Nginx

mysql

MySQL

postgresql

PostgreSQL

mongodb

MongoDB

couchdb

CouchDB

redis

Redis

aws

Amazon AWS

docker

Docker

django

Django

jenkins

Jenkins

haproxy

HAProxy

nagios

Nagios

cassandra

Cassandra

sinatra

Sinatra

varnish

Varnish

Using psymon

Without modifying your existing source code, you can instantly enable collection of known application logs to your application by using the psymon script to run your program. For example:

psymon --logs "/path/to/your/app --arg1 --arg2 --arg3"

See psymon man page for a complete list of options.

Using C/C++ SDK API

The Log Collection API can be accessed by including a single header file and linking the libpsyiage shared library. See Advanced Setup - Building your application with libpsyiage for details.

Real-time monitoring of known log files

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#include <psyiage/psyiagesdk.h>

int main(int argc, char **argv)
{
    psyiage_handle     pah;  // handle to psyiage instance

    /* ... Initialize code omitted for simplicity ... */

    // Search for known log files
    char**      logfiles = NULL;      // Array of known log files
    uint32_t*   logfileflags = NULL;  // Array of log flags
    int         numfiles = 0;         // Number of log files found

    rc = psyiage_log_glob_known_log_files(&logfiles, &logfileflags, &numfiles);

    // Monitor all log files found in real-time, at a period of 1 second
    int i;
    for (i = 0; i < numfiles; i++)
    {
        char *logfile = logfiles[i];
        rc = psyiage_log_monitor_log_file(pah, logfile, 1000);
    }

    // Free memory allocated to arrays
    psyiage_log_glob_free(logfiles, logfileflags, numfiles);

    /* ... Cleanup code omitted for simplicity ... */
}

Sample syslog JSON output

{
  "FilePath": "/var/log/messages",
  "Entries": [
    {
      "Duration": 5.006322384,
      "Timestamp": 1559440474.084455013,
      "TimeOfDay": "2019-06-02 10:54:34.084455013",
      "LogFile": "/var/log/messages",
      "LogText": "Jun  2 10:54:33 ubuntu log_monitor[35659]: psyiage 0.3.0 log monitoring test @ priority Warning"
    },
    {
      "Duration": 6.008110762,
      "Timestamp": 1559440475.086243391,
      "TimeOfDay": "2019-06-02 10:54:35.086243391",
      "LogFile": "/var/log/messages",
      "LogText": "Jun  2 10:54:34 ubuntu log_monitor[35659]: psyiage 0.3.0 log monitoring test @ priority Notice"
    },
    {
      "Duration": 6.008533001,
      "Timestamp": 1559440475.086665630,
      "TimeOfDay": "2019-06-02 10:54:35.086665630",
      "LogFile": "/var/log/messages",
      "LogText": "Jun  2 10:54:35 ubuntu log_monitor[35659]: psyiage 0.3.0 log monitoring test @ priority Info"
    }
  ]
},