OSQuery Client
dataBridges exposes powerful Client Function messaging service for devices and applications to implement high performance non-blocking request / response messages. Detailed API documentation
TODO
Project Source
Download source
Download the osQuery.client by clicking on this link
Download source
Download the osQuery.client by clicking on this link
Download source
Download the osQuery.client by clicking on this link
Download source
Download the osQuery.client by clicking on this link
Dependencies
Dependencies
For this application, you will require to use ip and databridges-sio-client-lib (dataBridges Client Library) with NodeJS.
- Install External dependencies
-
npm install ip --save
-
Install dataBridges library
npm install databridges-sio-client-lib --save
Download the package.json by clicking on this link
We will be using https://osquery.io/downloads for executing queries on target host. You need to download and install latest version from this site before running this code. Get the path of osqueryi installation and put in osQueryPath environment variable. We suggest to use environmental variable for security reasons.
Dependencies
For this application, you will require to use asyncio and databridges-sio-client-lib (dataBridges Client Library).
- Install External dependencies
-
pip3 install asyncio
-
Install dataBridges library
pip3 install databridges_sio_client_lib
We will be using https://osquery.io/downloads for executing queries on target host. You need to download and install latest version from this site before running this code. Get the path of osqueryi installation and put in osQueryPath environment variable. We suggest to use environmental variable for security reasons.
Dependencies
For this application, you will require to use Databridges.Sio.Client.Lib (dataBridges Client Library).
- Install dataBridges library from "Manage NuGet Packages..." from the Solution -> Application Name context menu.
We will be using https://osquery.io/downloads for executing queries on target host. You need to download and install latest version from this site before running this code. Get the path of osqueryi installation and put in osQueryPath environment variable. We suggest to use environmental variable for security reasons.
Download source
Set up a dataBridges account and app
Before we jump right into setting up an application with dataBridges, you’ll need to create a dataBridges account and app, if you don’t already have one:
-
Sign up for a dataBridges account.
-
Create a new app by selecting Apps and clicking Create New button.
-
You can retrieve your app credentials from the App Keys tab.
Preparing to run osQuery Client
Before you start, you need to make sure you have installed OSQuery from the official website and installed it on the client system. OSQuery is available for all major operating system. Once it is installed make a note of its installation binary path. Most of the times it is available in direct binary executable as osqueryi
.
Download the osQuery.client code from above given link and move it to application folder.
If you are preferring to use the environmental variable (preferred due to security reasons), you need to create 4 environment variables (dBridgeAuthURL, dBridgeAppKey, osQueryPath, deviceGroupName)
and store the required data. If you are not using environment variable, In the osQuery.client file you need to do few modification to store the keys and important data.
Property | Description |
---|---|
dBridgeAuthURL | (string) Client authentication url from dataBridges dashboard. |
dBridgeAppKey | (string) Client application Key from dataBridges dashboard. |
osQueryPath | (string) If osqueryi is not directly executable from any path, put full path of osqueryi binary. |
deviceGroupName | (string) Group Name against which this device is associated. |
Once the above changes is completed, save your code and your code is ready to be executed.
Note
You need to run osQueryClientAuthServer
application before starting this program.
Understanding important sections of osQuery Client source code.
Calling RPC osQueryClientAuthServer
for getting access_token
Define dbridge.access_token function. This is required because this app will need to subscribe to a sys:deviceGroupName to function as intended.
For the osQuery use case example, we will be running a separate Authentication server.This server will get request from each osQuery-client to get auth.token to subscribe to the deviceGroup.
The client will pass its hostName, IP, OS along with appKey that it has used to connect to dataBridges Network. Technically here you can replace or add more security information / tags that identifies the device as known.
For this example we will be using only host Name, IP and OS.
dbridge.access_token(async (channelName, sessionId, action, response) => {
let osPlatform = null;
switch (os.platform()) {
case 'win32':
osPlatform = 'win';
break;
case 'darwin':
osPlatform = 'macos';
break;
case 'linux':
osPlatform = 'linux';
break;
case 'sunos':
osPlatform = 'linux';
break;
case 'openbsd':
osPlatform = 'linux';
break;
case 'freebsd':
osPlatform = 'linux';
break;
default:
break;
}
if (osPlatform){
const jsonParam = { ch: channelName, sid: sessionId, act: action, hostname: os.hostname(), ip: ip.address(), clos: osPlatform, appkey: dbridge.appkey }
osQueryClientAuthRPC.call("getToken", JSON.stringify(jsonParam), 1000 * 60 * 2, null).then((res) => {
response.end(JSON.parse(res));
}).catch((err) => {
response.end({
statuscode: 1,
error_message: err.message,
accesskey: ''
});
console.log("AccessToken RPC Response", err.source, err.code, err.message);
});
} else {
console.log("Unable to retrive OS", os.platform());
response.end({
statuscode: 1,
error_message: "Unable to retrive OS",
accesskey: ''
});
}
});
self.dbridge.access_token(self.Get_Access_Token)
def getOs(self):
mysystem = platform.system()
if mysystem == 'Linux':
return 'linux'
elif mysystem == 'Windows':
return 'win'
elif mysystem == 'Darwin':
return 'macos'
else:
return ''
async def Get_Access_Token(self, channelName, sessionid, action, response):
try:
hostname = socket.gethostname()
IPAddr = socket.gethostbyname(hostname)
inparameter = json.dumps({"ch": channelName, "sid": sessionid, "act": action, "hostname": hostname, "ip": IPAddr, "clos": self.getOs() , "appkey": self.dbridge.appkey })
p = await self.osQueryClientAuthRPC.call("getToken" , inparameter, 10000, None )
def onResult(res):
try:
asyncio.create_task(response.end(json.loads(res)))
except Exception as e:
print(e)
def onError(res):
try:
print("AccessToken RPC Response", res.source, res.code, res.message)
asyncio.create_task(response.exception("1" , res.message))
except Exception as e:
print(e)
p.then(onResult).catch(onError)
except Exception as e:
print(e)
Action<object, object, object, object> iGetAccessToken = this.GetToken;
this.dbridge.access_token(iGetAccessToken);
public async void GetToken(object channelname, object sessionid, object action, object response) {
string m_channelname = channelname as string;
string m_sessionid = sessionid as string;
string m_action = action as string;
CprivateResponse m_response = response as CprivateResponse;
// Define payload structure
Dictionary<string, object> payLoad = new Dictionary<string, object> {
{ "ch", channelname } ,
{ "sid", sessionid } ,
{ "act" , action },
{ "hostname" , Dns.GetHostName() },
{ "ip", GetLocalIPAddress() },
{ "clos", GetOS() },
{ "appkey" , this.dbridge.appkey }
};
int i = 0;
while (!this.osQueryClientAuthRPC.isOnline() && i < 3) {
await Task.Delay(1000);
i++;
}
if (this.osQueryClientAuthRPC != null && this.osQueryClientAuthRPC.isOnline()) {
string data = JsonConvert.SerializeObject(payLoad);//
Action<object> ipcprogress;
IPromise<object> promise = await osQueryClientAuthRPC.call("getToken", data, 1000 * 60 * 2, null);
promise.Then(async (result) => {
dynamic brs_object = JsonConvert.DeserializeObject<dynamic>(result as string);
int statuscode = Convert.ToInt32(brs_object["statuscode"]);
string error_message = brs_object["error_message"] as string;
string accesskey = brs_object["accesskey"];
CPrivateInfo privateinfo = new CPrivateInfo(statuscode, error_message, accesskey);
await m_response.end(privateinfo);
})
.Catch(async (exec) => {
int statuscode = 1;
string error_message = "Exception raised";
string accesskey = "";
CPrivateInfo privateinfo = new CPrivateInfo(statuscode, error_message, accesskey);
await m_response.end(privateinfo);
});
} else {
Console.WriteLine("osQueryClientAuthRPC offline");
}
}
public string GetLocalIPAddress() {
var host = Dns.GetHostEntry(Dns.GetHostName());
foreach (var ip in host.AddressList) {
if (ip.AddressFamily == AddressFamily.InterNetwork) {
return ip.ToString();
}
}
throw new Exception("No network adapters with an IPv4 address in the system!");
}
public string GetOS() {
OperatingSystem os = Environment.OSVersion;
PlatformID pid = os.Platform;
switch (pid) {
case PlatformID.Win32NT:
return "win";
case PlatformID.Win32S:
return "win";
case PlatformID.Win32Windows:
return "win";
case PlatformID.WinCE:
return "win";
case PlatformID.Unix:
return "linux";
case PlatformID.MacOSX:
return "macos";
}
return "unknown";
}
Enable Client Functions and defining functions for osQuery Execution.
To enable remote execution of client functions in dataBridges network, you need to enable dbridge.cf.enable
property. You can then define functions to be exposed using client function in dbridge.cf.functions
method. In our current application we have exposed osQueryExecutor
which will exeucte the query sent by the Caller and send back results as JSON.
// Enable client function
dbridge.cf.enable = true;
/*
exposed client function which execute the osqueryi with query recieved by from the caller
caller will send the json stringfy parameter
{"format": "" , "query": "SELECT name, path, pid FROM processes WHERE on_disk = 0;"}
response format
{"returncode": 0 or any number , "osqueryresult": executed query output}
*/
function osQueryExecutor(payload, response) {
console.log('query received > ', payload);
inparameter = JSON.parse(payload);
// execute the osqueryi process with query parameter
exec(osQueryPath + " " + inparameter.query, (error, stdout, stderr) => {
outparameter = {}
// return the exception if any error occurred
if (error) {
console.error('error: ', error.message);
response.exception("error", error.message)
return;
}
// return the exception if any stderr occurred
if (stderr) {
console.error('stderr: ', stderr);
response.exception("error", stderr)
return;
}
outparameter = { "osqueryresult": stdout, "returncode": 0 };
console.log('osqueryresult: ', stdout);
//return the result
response.end(JSON.stringify(outparameter));
});
}
// Enable clientFuntion before exposing function and dataBridges connection.
dbridge.cf.functions = function () {
try {
// Bind the exposed function, so that library can register it for receiving calls.
this.regfn("osquery", osQueryExecutor);
} catch (err) {
console.log(err)
}
}
# Enable client funtion.
self.dbridge.cf.enable = True
self.dbridge.cf.functions = self.client_functions
# Enable clientFuntion before exposing function and dataBridges connection.
async def client_functions(self):
try:
# Bind the exposed function, so that library can register it for receiving calls.
self.dbridge.cf.regfn("osquery" , self.osquery_Executor)
except Exception as e:
print(e)
async def osquery_Executor(self, inputparameter , response):
try:
response.tracker = True
inparameter = json.loads(inputparameter)
format = inparameter["format"]
query = inparameter["query"]
# execute the osqueryi process with query parameter
if format:
process = subprocess.Popen([self.osquery_path, format, query],
stdout=subprocess.PIPE,
universal_newlines=True)
else:
process = subprocess.Popen([self.osquery_path, query],
stdout=subprocess.PIPE,
universal_newlines=True)
cf_output = ""
cf_returncode =None
while True:
output = process.stdout.readline()
cf_output = cf_output + output.strip() + "\n"
return_code = process.poll()
if return_code is not None:
# Process has finished, read rest of the output
for output in process.stdout.readlines():
cf_output = cf_output + output.strip()
cf_returncode = return_code
break
cf_data = {"osqueryresult": cf_output , "returncode": cf_returncode }
await response.end(json.dumps(cf_data))
except Exception as e:
print(e)
await response.exception("error" , e)
this.dbridge.cf.enable = true;
Action<object> ifunctions = this.cf_function;
this.dbridge.cf.functions = ifunctions;
// Enable clientFuntion before exposing function and dataBridges connection.
public void cf_function(object sender) {
try {
if (sender.GetType().Equals(typeof(dBridges.clientFunctions.cfclient))) {
Action<object, object> iosquery = this.osQueryExecutor;
try {
// Bind the exposed function, so that library can register it for receiving calls.
(sender as dBridges.clientFunctions.cfclient).regfn("osquery", iosquery);
} catch (dBError e) {
Console.WriteLine(" {0} , {1} , {2} ", e.source, e.code, e.message);
}
}
} catch (Exception e) {
Console.WriteLine(e);
}
}
public async void osQueryExecutor(object inparam, object response) {
Console.WriteLine("inparameter {0}", inparam);
dBridges.responseHandler.CResponseHandler responsehandler = response as dBridges.responseHandler.CResponseHandler;
dynamic jinparam = JsonConvert.DeserializeObject<dynamic>(inparam as string);
var process = new Process {
StartInfo = new ProcessStartInfo {
FileName = this.osQueryPath,
Arguments = (string)jinparam["query"],
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
}
};
process.Start();
string line = "";
while (!process.StandardOutput.EndOfStream) {
line = line + process.StandardOutput.ReadLine() + Environment.NewLine;
}
process.WaitForExit();
Console.WriteLine(line);
Dictionary<string, string> ireturn = new Dictionary<string, string>();
ireturn["returncode"] = "0";
ireturn["osqueryresult"] = line;
await responsehandler.end(JsonConvert.SerializeObject(ireturn));
}
dataBridges Network connection and reconnection.
To make sure the dataBridges network connection is always connected, we will be tweaking few porperties and code. Application will try to do reconnection using dataBridges auto reconnection process for 500 times as defined in the property. Also after disconnection from the network (when reconnection fails) we will be calling the connection process again. This will do a infinite try till it get successfull connection.
dbridge.autoReconnect = true;
dbridge.maxReconnectionRetries = 500;
const connectTodataBridgesNetwork = () => {
// Connect to dataBridges. If any runtime error it will be caught in catch().
dbridge.connect().catch((err) => {
console.log('dBridge Connection exception..', err);
});
}
//Bind to disconnected event, to get intimation about dataBridges network disconnection. Do note that we have configured dataBridges client to retry 500 times post disconnection. This event will be encountered only after 500 retries. When that happens, we will reconnect back to dataBridges again.
dbridge.connectionstate.bind("disconnected", () => {
console.log('dataBridges network', 'disconnected');
// Attempt reconnect on disconnection.
osQueryClientAuthRPC = null;
deviceGroup = null;
connectTodataBridgesNetwork();
});
connectTodataBridgesNetwork();
self.dbridge.autoReconnect = True
self.dbridge.maxReconnectionRetries = 500
async def connectTodataBridgesNetwork(self):
# Connect to dataBridges. If any runtime error it will be caught in Exception.
try:
await asyncio.sleep(0.0001)
await self.dbridge.connect()
except Exception as e:
print(e)
# Bind to disconnected event, to get intimation about dataBridges network disconnection. Do note that we have configured dataBridges client to retry 500 times post disconnection. This event will be encountered only after 500 retries. When that happens, we will reconnect back to dataBridges again.
async def disconnected(self):
try:
print("dataBridges network", "disconnected")
# Attempt reconnect on disconnection.
await self.connectTodataBridgesNetwork()
except Exception as e:
print(e)
await self.connectTodataBridgesNetwork()
this.dbridge.maxReconnectionRetries = 500;
this.dbridge.autoReconnect = true;
public async Task connectTodataBridgesNetwork() {
try {
await this.dbridge.connect();
} catch (dBError dberror) {
Console.WriteLine("dBridge connect exception..{0} , {1} , {2}", dberror.source, dberror.code, dberror.message);
}
}
//Bind to disconnected event, to get intimation about dataBridges network disconnection. Do note that we have configured dataBridges client to retry 500 times post disconnection. This event will be encountered only after 500 retries. When that happens, we will reconnect back to dataBridges again.
Action<object> disconnected_callback;
this.dbridge.connectionstate.bind("disconnected", disconnected_callback = async (object data) => {
Console.WriteLine("dataBridges network", "disconnected");
await connectTodataBridgesNetwork();
});
await connectTodataBridgesNetwork();
dataBridges on Connection process.
Once the dataBridge network is successfully connected, We will be first connecting to osQueryClientAuth
RPC server for making sure whenever trusted_token request is made, the connection is available. On sucessfull connection of RPC server, the application will subscribe to sys:deviceGroupName
channel to mark its presence and to inform the osQuery.queryScript
application that it is available for serving osQuery execution.
//Bind to connected event.
dbridge.connectionstate.bind("connected", () => {
console.log('dataBridges network', 'connected');
osQueryClientAuthRPC = dbridge.rpc.connect("osQueryClientAuth");
// Bind to rpc.server.connect.success event
osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.success", async (payload, metadata) => {
console.log('osQueryClientAuthRPC successfully connected.')
// Subscribe to channel 'sys:' + deviceGroupName,
deviceGroup = await dbridge.channel.subscribe("sys:" + deviceGroupName);
// Bind to subscribe.success event
deviceGroup.bind("dbridges:subscribe.success", (payload, metadata) => {
console.log('Subscribed to :', 'sys:' + deviceGroupName)
});
// Bind to subscribe.fail event to understand if any issues in subscription.
deviceGroup.bind("dbridges:subscribe.fail", (payload, metadata) => {
console.log('Subscription failed for ', 'sys:' + deviceGroupName, payload.source, payload.code, payload.message)
});
});
// Bind to rpc.server.connect.fail event
osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.fail", async (payload, metadata) => {
console.log("osQueryClientAuthRPC Server connection failed :", payload.code, payload.source, payload.message)
});
});
# Bind to connected event.
async def connected(self):
try:
print("dataBridges network", "connected")
self.osQueryClientAuthRPC = await self.dbridge.rpc.connect("osQueryClientAuth")
# Bind to rpc.server.connect.success event
self.osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.success", self.server_connect_success)
# Bind to rpc.server.connect.fail event
self.osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.fail", self.server_connect_fail)
except Exception as e:
print(e)
# Bind to rpc.server.connect.success event
async def server_connect_success(self , payload , metadata):
try:
print("osQueryClientAuthRPC successfully connected.")
# Subscribe to channel 'sys:' + deviceGroupName,
self.deviceGroup = await self.dbridge.channel.subscribe("sys:" + self.deviceGroupName)
# Bind to subscribe.success event
self.deviceGroup.bind("dbridges:subscribe.success", self.subscribe_success)
# Bind to subscribe.fail event
self.deviceGroup.bind("dbridges:subscribe.fail", self.subscribe_fail)
except Exception as e:
print(e)
# Bind to rpc.server.connect.fail event
async def server_connect_fail(self, payload , metadata):
print("osQueryClientAuthRPC Server connection failed :", payload.code, payload.source, payload.message)
# Bind to subscribe.success event
async def subscribe_success(self, payload , metadata):
print("Subscribed to :", "sys:" + self.deviceGroupName)
# Bind to subscribe.fail event to understand if any issues in subscription.
async def subscribe_fail(self, payload, metadata):
print("Subscription failed for sys:" + self.deviceGroupName, payload.source, payload.code, payload.message)
Action<object> connected_callback;
//Bind to connected event.
this.dbridge.connectionstate.bind("connected", connected_callback = async (object data) => {
Console.WriteLine("dataBridges network", "connected");
try {
this.osQueryClientAuthRPC = await dbridge.rpc.connect("osQueryClientAuth");
// Bind to rpc.server.connect.success event
Action<object, object> iaccesstoken;
osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.success", iaccesstoken = async (object payload, object metadata) => {
// Subscribe to channel "sys:" + deviceGroupName,
this.deviceGroup = await this.dbridge.channel.subscribe("sys:" + this.deviceGroupName);
Action<object, object> a_connectsuccess;
// Bind to subscribe.success event
this.deviceGroup.bind("dbridges:subscribe.success", a_connectsuccess = (object sub_payload, object sub_metadata) => {
dBridges.Utils.metaData mdt = sub_metadata as dBridges.Utils.metaData;
Console.WriteLine("Subscribed to : sys:{0}" , this.deviceGroupName);
});
Action<object, object> a_connectfail;
// Bind to subscribe.fail event to understand if any issues in subscription.
this.deviceGroup.bind("dbridges:subscribe.fail", a_connectfail = (object sub_payload, object sub_metadata) => {
dBridges.exceptions.dBError mdt = sub_metadata as dBridges.exceptions.dBError;
Console.WriteLine("Subscription failed for \"sys:\"{0} {1}{2}{3}", this.deviceGroupName, mdt.source, mdt.code, mdt.message);
});
});
// Bind to rpc.server.connect.fail event
Action<object, object> iaccessfail;
this.osQueryClientAuthRPC.bind("dbridges:rpc.server.connect.fail", iaccessfail = (object payload, object metadata) => {
dBridges.exceptions.dBError mdt = payload as dBridges.exceptions.dBError;
Console.WriteLine("osQueryClientAuthRPC Server connection failed :", mdt.code, mdt.source, mdt.message);
});
} catch (dBError dberror) {
Console.WriteLine("dBridge rpc event bind exception..{0} , {1} , {2}", dberror.source, dberror.code, dberror.message);
}
});