Skip to content

OSQuery Client

dataBridges exposes powerful Client Function messaging service for devices and applications to implement high performance non-blocking request / response messages. Detailed API documentation

TODO

Project Source

Download source

Download the osQuery.client by clicking on this link

Download source

Download the osQuery.client by clicking on this link

Download source

Download the osQuery.client by clicking on this link

Download source

Download the osQuery.client by clicking on this link

Dependencies

Dependencies

For this application, you will require to use ip and databridges-sio-client-lib (dataBridges Client Library) with NodeJS.

  • Install External dependencies
  • npm install ip --save

  • Install dataBridges library

  • npm install databridges-sio-client-lib --save

Download the package.json by clicking on this link

We will be using https://osquery.io/downloads for executing queries on target host. You need to download and install latest version from this site before running this code. Get the path of osqueryi installation and put in osQueryPath environment variable. We suggest to use environmental variable for security reasons.

const osquery_path = process.env.osQueryPath;

Dependencies

For this application, you will require to use asyncio and databridges-sio-client-lib (dataBridges Client Library).

  • Install External dependencies
  • pip3 install asyncio

  • Install dataBridges library

  • pip3 install databridges_sio_client_lib

We will be using https://osquery.io/downloads for executing queries on target host. You need to download and install latest version from this site before running this code. Get the path of osqueryi installation and put in osQueryPath environment variable. We suggest to use environmental variable for security reasons.

self.osquery_path = os.environ["osQueryPath"]

Dependencies

For this application, you will require to use Databridges.Sio.Client.Lib (dataBridges Client Library).

  • Install dataBridges library from "Manage NuGet Packages..." from the Solution -> Application Name context menu.

We will be using https://osquery.io/downloads for executing queries on target host. You need to download and install latest version from this site before running this code. Get the path of osqueryi installation and put in osQueryPath environment variable. We suggest to use environmental variable for security reasons.

this.osQueryPath = Environment.GetEnvironmentVariable("osQueryPath");

Download source

Set up a dataBridges account and app

Before we jump right into setting up an application with dataBridges, you’ll need to create a dataBridges account and app, if you don’t already have one:

  1. Sign up for a dataBridges account.

  2. Create a new app by selecting Apps and clicking Create New button.

  3. You can retrieve your app credentials from the App Keys tab.

Preparing to run osQuery Client

Before you start, you need to make sure you have installed OSQuery from the official website and installed it on the client system. OSQuery is available for all major operating system. Once it is installed make a note of its installation binary path. Most of the times it is available in direct binary executable as osqueryi.

Download the osQuery.client code from above given link and move it to application folder.

If you are preferring to use the environmental variable (preferred due to security reasons), you need to create 4 environment variables (dBridgeAuthURL, dBridgeAppKey, osQueryPath, deviceGroupName) and store the required data. If you are not using environment variable, In the osQuery.client file you need to do few modification to store the keys and important data.

Property Description
dBridgeAuthURL (string) Client authentication url from dataBridges dashboard.
dBridgeAppKey (string) Client application Key from dataBridges dashboard.
osQueryPath (string) If osqueryi is not directly executable from any path, put full path of osqueryi binary.
deviceGroupName (string) Group Name against which this device is associated.

Once the above changes is completed, save your code and your code is ready to be executed.

Note

You need to run osQueryClientAuthServer application before starting this program.

Understanding important sections of osQuery Client source code.

Calling RPC osQueryClientAuthServer for getting access_token

Define dbridge.access_token function. This is required because this app will need to subscribe to a sys:deviceGroupName to function as intended.

For the osQuery use case example, we will be running a separate Authentication server.This server will get request from each osQuery-client to get auth.token to subscribe to the deviceGroup.

The client will pass its hostName, IP, OS along with appKey that it has used to connect to dataBridges Network. Technically here you can replace or add more security information / tags that identifies the device as known.

For this example we will be using only host Name, IP and OS.

dbridge.access_token(async (channelName, sessionId, action, response) => {
    let osPlatform = null;
    switch (os.platform()) {
        case 'win32':
            osPlatform = 'win';
            break;
        case 'darwin':
            osPlatform = 'macos';
            break;
        case 'linux':
            osPlatform = 'linux';
            break;
        case 'sunos':
            osPlatform = 'linux';
            break;
        case 'openbsd':
            osPlatform = 'linux';
            break;
        case 'freebsd':
            osPlatform = 'linux';
            break;
        default: 
            break;
    }
    if (osPlatform){
        const jsonParam = { ch: channelName, sid: sessionId, act: action, hostname: os.hostname(), ip: ip.address(), clos: osPlatform, appkey: dbridge.appkey }
        osQueryClientAuthRPC.call("getToken", JSON.stringify(jsonParam), 1000 * 60 * 2, null).then((res) => {
            response.end(JSON.parse(res));
        }).catch((err) => {
            response.end({
                statuscode: 1,
                error_message: err.message,
                accesskey: ''
            });
            console.log("AccessToken RPC Response", err.source, err.code, err.message);
        }); 
    } else {
        console.log("Unable to retrive OS", os.platform());
        response.end({
            statuscode: 1,
            error_message: "Unable to retrive OS",
            accesskey: ''
        });
    }
});
self.dbridge.access_token(self.Get_Access_Token)


def getOs(self):
    mysystem = platform.system() 
    if mysystem == 'Linux':
        return  'linux'
    elif mysystem == 'Windows':
        return  'win'
    elif mysystem == 'Darwin':
        return  'macos'
    else:
        return ''

async  def Get_Access_Token(self, channelName, sessionid, action, response):
    try:
        hostname = socket.gethostname()
        IPAddr = socket.gethostbyname(hostname)
        inparameter = json.dumps({"ch": channelName, "sid": sessionid, "act": action, "hostname": hostname,  "ip": IPAddr, "clos": self.getOs() , "appkey": self.dbridge.appkey })

        p = await self.osQueryClientAuthRPC.call("getToken" , inparameter,  10000, None )

        def onResult(res):
            try:
                asyncio.create_task(response.end(json.loads(res)))
            except Exception as e:
                print(e)

        def onError(res):
            try: 
                print("AccessToken RPC Response", res.source, res.code, res.message)
                asyncio.create_task(response.exception("1" ,  res.message))
            except Exception as e:
                print(e)
        p.then(onResult).catch(onError)

    except Exception as e:
        print(e)
Action<object, object, object, object> iGetAccessToken = this.GetToken;
this.dbridge.access_token(iGetAccessToken);

public async void GetToken(object channelname, object sessionid, object action, object response) {
    string m_channelname = channelname as string;
    string m_sessionid = sessionid as string;
    string m_action = action as string;
    CprivateResponse m_response = response as CprivateResponse;
    // Define payload structure
    Dictionary<string, object> payLoad = new Dictionary<string, object> {
        { "ch", channelname } ,
        { "sid", sessionid } ,
        { "act" ,  action },
        { "hostname" , Dns.GetHostName()  },
        { "ip", GetLocalIPAddress() },
        { "clos", GetOS()  },
        { "appkey" , this.dbridge.appkey }
    };
    int i = 0;
    while (!this.osQueryClientAuthRPC.isOnline() && i < 3) {
        await Task.Delay(1000);
        i++;
    }

    if (this.osQueryClientAuthRPC != null && this.osQueryClientAuthRPC.isOnline()) { 
        string data = JsonConvert.SerializeObject(payLoad);//
        Action<object> ipcprogress;
        IPromise<object> promise = await osQueryClientAuthRPC.call("getToken", data, 1000 * 60 * 2, null);

        promise.Then(async (result) => {
            dynamic brs_object = JsonConvert.DeserializeObject<dynamic>(result as string);

            int statuscode = Convert.ToInt32(brs_object["statuscode"]);
            string error_message = brs_object["error_message"] as string;
            string accesskey = brs_object["accesskey"];
            CPrivateInfo privateinfo = new CPrivateInfo(statuscode, error_message, accesskey);
            await m_response.end(privateinfo);
        })
        .Catch(async (exec) => {
            int statuscode = 1;
            string error_message = "Exception raised";
            string accesskey = "";
            CPrivateInfo privateinfo = new CPrivateInfo(statuscode, error_message, accesskey);
            await m_response.end(privateinfo);
        });
    } else {
        Console.WriteLine("osQueryClientAuthRPC offline");
    }
}

public  string GetLocalIPAddress() {
    var host = Dns.GetHostEntry(Dns.GetHostName());
    foreach (var ip in host.AddressList) {
        if (ip.AddressFamily == AddressFamily.InterNetwork) {
            return ip.ToString();
        }
    }
    throw new Exception("No network adapters with an IPv4 address in the system!");
}

public string GetOS() {
    OperatingSystem os = Environment.OSVersion;
    PlatformID pid = os.Platform;
    switch (pid) {
        case PlatformID.Win32NT:
            return "win";
        case PlatformID.Win32S:
            return "win";
        case PlatformID.Win32Windows:
            return "win";
        case PlatformID.WinCE:
            return "win";
        case PlatformID.Unix:
            return "linux";
        case PlatformID.MacOSX:
            return "macos";
    }
    return "unknown";
}

Enable Client Functions and defining functions for osQuery Execution.

To enable remote execution of client functions in dataBridges network, you need to enable dbridge.cf.enable property. You can then define functions to be exposed using client function in dbridge.cf.functions method. In our current application we have exposed osQueryExecutor which will exeucte the query sent by the Caller and send back results as JSON.

// Enable client function 
dbridge.cf.enable = true;
/* 
 exposed client function which execute the osqueryi with query recieved by from the caller
 caller will send the json stringfy parameter 
    {"format": "" , "query":  "SELECT name, path, pid FROM processes WHERE on_disk = 0;"}
    response format
    {"returncode": 0 or any number ,  "osqueryresult": executed query output}
 */ 
function osQueryExecutor(payload, response) {
    console.log('query received > ', payload);
    inparameter = JSON.parse(payload);
    // execute the osqueryi process with query parameter
    exec(osQueryPath + " " + inparameter.query, (error, stdout, stderr) => {
        outparameter = {}
        // return the exception if any error occurred
        if (error) {
            console.error('error: ', error.message);
            response.exception("error", error.message)
            return;
        }
        // return the exception if any stderr occurred
        if (stderr) {
            console.error('stderr: ', stderr);
            response.exception("error", stderr)
            return;
        }
        outparameter = { "osqueryresult": stdout, "returncode": 0 };
        console.log('osqueryresult: ', stdout);
        //return the result 
        response.end(JSON.stringify(outparameter));
    });
}

// Enable clientFuntion before exposing function and dataBridges connection.
dbridge.cf.functions = function () {
    try {
        // Bind the exposed function, so that library can register it for receiving calls.
        this.regfn("osquery", osQueryExecutor);
    } catch (err) {
        console.log(err)
    }
}
# Enable client funtion.
self.dbridge.cf.enable = True

self.dbridge.cf.functions = self.client_functions

# Enable clientFuntion before exposing function and dataBridges connection.
async  def client_functions(self):
    try:
        # Bind the exposed function, so that library can register it for receiving calls.
        self.dbridge.cf.regfn("osquery" ,  self.osquery_Executor)
    except Exception as e:
        print(e)

async  def osquery_Executor(self, inputparameter , response):
    try:
        response.tracker = True
        inparameter = json.loads(inputparameter)
        format = inparameter["format"]
        query =  inparameter["query"]
        #  execute the osqueryi process with query parameter
        if format:
            process = subprocess.Popen([self.osquery_path, format, query],
                                       stdout=subprocess.PIPE,
                                       universal_newlines=True)
        else:
            process = subprocess.Popen([self.osquery_path, query],
                                       stdout=subprocess.PIPE,
                                       universal_newlines=True)
        cf_output = ""
        cf_returncode =None
        while True:
            output = process.stdout.readline()
            cf_output = cf_output + output.strip() + "\n"
            return_code = process.poll()
            if return_code is not None:
                # Process has finished, read rest of the output
                for output in process.stdout.readlines():
                    cf_output = cf_output +   output.strip()
                cf_returncode = return_code
                break
        cf_data = {"osqueryresult": cf_output , "returncode": cf_returncode }
        await response.end(json.dumps(cf_data))
    except Exception as e:
        print(e)
        await response.exception("error" , e)
this.dbridge.cf.enable = true;
Action<object> ifunctions = this.cf_function;
this.dbridge.cf.functions = ifunctions;

// Enable clientFuntion before exposing function and dataBridges connection.
public void cf_function(object sender) {
    try {
        if (sender.GetType().Equals(typeof(dBridges.clientFunctions.cfclient))) {
            Action<object, object> iosquery = this.osQueryExecutor;
            try {
                // Bind the exposed function, so that library can register it for receiving calls.
                (sender as dBridges.clientFunctions.cfclient).regfn("osquery", iosquery);
            } catch (dBError e) {
                Console.WriteLine(" {0} , {1} , {2} ", e.source, e.code, e.message);
            }
        }
    } catch (Exception e) {
        Console.WriteLine(e);
    }
}

public async void osQueryExecutor(object inparam, object response) {
    Console.WriteLine("inparameter {0}", inparam);
    dBridges.responseHandler.CResponseHandler responsehandler = response as dBridges.responseHandler.CResponseHandler;

    dynamic jinparam = JsonConvert.DeserializeObject<dynamic>(inparam as string);  
    var process = new Process {
        StartInfo = new ProcessStartInfo {
            FileName = this.osQueryPath,
            Arguments = (string)jinparam["query"],
            UseShellExecute = false,
            RedirectStandardOutput = true,
            CreateNoWindow = true
        }
    };

    process.Start();
    string line = "";
    while (!process.StandardOutput.EndOfStream) {
        line = line + process.StandardOutput.ReadLine() + Environment.NewLine;
    }
    process.WaitForExit();
    Console.WriteLine(line);
    Dictionary<string, string> ireturn = new Dictionary<string, string>();
    ireturn["returncode"] = "0";
    ireturn["osqueryresult"] = line;
    await responsehandler.end(JsonConvert.SerializeObject(ireturn));
}

dataBridges Network connection and reconnection.

To make sure the dataBridges network connection is always connected, we will be tweaking few porperties and code. Application will try to do reconnection using dataBridges auto reconnection process for 500 times as defined in the property. Also after disconnection from the network (when reconnection fails) we will be calling the connection process again. This will do a infinite try till it get successfull connection.

dbridge.autoReconnect = true;
dbridge.maxReconnectionRetries = 500;

const connectTodataBridgesNetwork = () => {
    // Connect to dataBridges. If any runtime error it will be caught in catch().
    dbridge.connect().catch((err) => {
        console.log('dBridge Connection exception..', err);
    });
}

//Bind to disconnected event, to get intimation about dataBridges network disconnection. Do note that we have configured dataBridges client to retry 500 times post disconnection. This event will be encountered only after 500 retries. When that happens, we will reconnect back to dataBridges again.
dbridge.connectionstate.bind("disconnected", () => {
    console.log('dataBridges network', 'disconnected');

    // Attempt reconnect on disconnection.
    osQueryClientAuthRPC = null;
    deviceGroup = null;
    connectTodataBridgesNetwork();
});

connectTodataBridgesNetwork();
self.dbridge.autoReconnect = True
self.dbridge.maxReconnectionRetries = 500

async def connectTodataBridgesNetwork(self):
    # Connect to dataBridges. If any runtime error it will be caught in Exception.
    try:
        await asyncio.sleep(0.0001)
        await self.dbridge.connect()
    except Exception as e:
        print(e)

# Bind to disconnected event, to get intimation about dataBridges network disconnection. Do note that we have configured dataBridges client to retry 500 times post disconnection. This event will be encountered only after 500 retries. When that happens, we will reconnect back to dataBridges again.
async def disconnected(self):
    try:
        print("dataBridges network", "disconnected")

        # Attempt reconnect on disconnection.
        await self.connectTodataBridgesNetwork()
    except Exception as e:
        print(e)

await self.connectTodataBridgesNetwork()
this.dbridge.maxReconnectionRetries = 500;
this.dbridge.autoReconnect = true;

public async Task connectTodataBridgesNetwork() {
    try {
        await this.dbridge.connect();

    } catch (dBError dberror) {
        Console.WriteLine("dBridge connect exception..{0} , {1} , {2}", dberror.source, dberror.code, dberror.message);
    }
}

//Bind to disconnected event, to get intimation about dataBridges network disconnection. Do note that we have configured dataBridges client to retry 500 times post disconnection. This event will be encountered only after 500 retries. When that happens, we will reconnect back to dataBridges again.
Action<object> disconnected_callback;
this.dbridge.connectionstate.bind("disconnected", disconnected_callback = async (object data) => {
    Console.WriteLine("dataBridges network", "disconnected");
    await connectTodataBridgesNetwork();
});

await connectTodataBridgesNetwork();

dataBridges on Connection process.

Once the dataBridge network is successfully connected, We will be first connecting to osQueryClientAuth RPC server for making sure whenever trusted_token request is made, the connection is available. On sucessfull connection of RPC server, the application will subscribe to sys:deviceGroupName channel to mark its presence and to inform the osQuery.queryScript application that it is available for serving osQuery execution.

//Bind to connected event.
dbridge.connectionstate.bind("connected", () => {
    console.log('dataBridges network', 'connected');

    osQueryClientAuthRPC = dbridge.rpc.connect("osQueryClientAuth");

    // Bind to rpc.server.connect.success event
    osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.success", async (payload, metadata) => {
        console.log('osQueryClientAuthRPC successfully connected.')
        // Subscribe to channel 'sys:' + deviceGroupName,
        deviceGroup = await dbridge.channel.subscribe("sys:" + deviceGroupName);

        // Bind to subscribe.success event
        deviceGroup.bind("dbridges:subscribe.success", (payload, metadata) => {
            console.log('Subscribed to :', 'sys:' + deviceGroupName)
        });

        // Bind to subscribe.fail event to understand if any issues in subscription.
        deviceGroup.bind("dbridges:subscribe.fail", (payload, metadata) => {
            console.log('Subscription failed for ', 'sys:' + deviceGroupName, payload.source, payload.code, payload.message)
        });
    });

    // Bind to rpc.server.connect.fail event
    osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.fail", async (payload, metadata) => {
        console.log("osQueryClientAuthRPC Server connection failed :", payload.code, payload.source, payload.message)
    });

});
# Bind to connected event.
async def connected(self):
    try:
        print("dataBridges network", "connected")

        self.osQueryClientAuthRPC = await self.dbridge.rpc.connect("osQueryClientAuth")
        # Bind to rpc.server.connect.success event
        self.osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.success", self.server_connect_success)
        # Bind to rpc.server.connect.fail event
        self.osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.fail", self.server_connect_fail)

    except Exception as e:
        print(e)

# Bind to rpc.server.connect.success event
async def server_connect_success(self , payload ,  metadata):
    try:
        print("osQueryClientAuthRPC successfully connected.")

        # Subscribe to channel 'sys:' + deviceGroupName,
        self.deviceGroup = await self.dbridge.channel.subscribe("sys:" +  self.deviceGroupName)

        # Bind to subscribe.success event
        self.deviceGroup.bind("dbridges:subscribe.success", self.subscribe_success)
        # Bind to subscribe.fail event
        self.deviceGroup.bind("dbridges:subscribe.fail", self.subscribe_fail)

    except Exception as e:
        print(e)

# Bind to rpc.server.connect.fail event
async def server_connect_fail(self, payload , metadata):
    print("osQueryClientAuthRPC Server connection failed :", payload.code, payload.source, payload.message)

# Bind to subscribe.success event
async  def subscribe_success(self, payload , metadata):
    print("Subscribed to :", "sys:" + self.deviceGroupName)

# Bind to subscribe.fail event to understand if any issues in subscription.
async def subscribe_fail(self, payload, metadata):
    print("Subscription failed for sys:" +  self.deviceGroupName, payload.source, payload.code, payload.message)
Action<object> connected_callback;
//Bind to connected event.
this.dbridge.connectionstate.bind("connected", connected_callback = async (object data) => {
    Console.WriteLine("dataBridges network", "connected");

    try {
        this.osQueryClientAuthRPC = await dbridge.rpc.connect("osQueryClientAuth");
        // Bind to rpc.server.connect.success event
        Action<object, object> iaccesstoken;
        osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.success", iaccesstoken = async (object payload, object metadata) => {
            // Subscribe to channel "sys:" + deviceGroupName,
            this.deviceGroup = await this.dbridge.channel.subscribe("sys:" + this.deviceGroupName);

            Action<object, object> a_connectsuccess;
            // Bind to subscribe.success event
            this.deviceGroup.bind("dbridges:subscribe.success", a_connectsuccess =  (object sub_payload, object sub_metadata) => {
                dBridges.Utils.metaData mdt = sub_metadata as dBridges.Utils.metaData;
                Console.WriteLine("Subscribed to : sys:{0}" , this.deviceGroupName);
            });

            Action<object, object> a_connectfail;
            // Bind to subscribe.fail event to understand if any issues in subscription.
            this.deviceGroup.bind("dbridges:subscribe.fail", a_connectfail = (object sub_payload, object sub_metadata) => {
                dBridges.exceptions.dBError mdt = sub_metadata as dBridges.exceptions.dBError;
                Console.WriteLine("Subscription failed for \"sys:\"{0} {1}{2}{3}", this.deviceGroupName, mdt.source, mdt.code, mdt.message);
            });
        });

        // Bind to rpc.server.connect.fail event
        Action<object, object> iaccessfail;
        this.osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.fail", iaccessfail = (object payload, object metadata) => {
            dBridges.exceptions.dBError mdt = payload as dBridges.exceptions.dBError;
            Console.WriteLine("osQueryClientAuthRPC Server connection failed :", mdt.code, mdt.source, mdt.message);
        });
    } catch (dBError dberror) {
        Console.WriteLine("dBridge rpc event bind exception..{0} , {1} , {2}", dberror.source, dberror.code, dberror.message);
    }
});